Multi-process interactive systems and methods

ABSTRACT

A multi-process interactive system is described. The system includes numerous processes running on a processing device. The processes include separable program execution contexts of application programs, such that each application program comprises at least one process. The system translates events of each process into data capsules. A data capsule includes an application-independent representation of event data of an event and state information of the process originating the content of the data capsule. The system transfers the data messages into pools or repositories. Each process operates as a recognizing process, where the recognizing process recognizes in the pools data capsules comprising content that corresponds to an interactive function of the recognizing process and/or an identification of the recognizing process. The recognizing process retrieves recognized data capsules from the pools and executes processing appropriate to contents of the recognized data capsules.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/875,661, filed 19 Jan. 2018, which is a continuation of U.S. patentapplication Ser. No. 14/733,125, filed 8 Jun. 2015, which is acontinuation of U.S. patent application Ser. No. 12/579,340, filed 14Oct. 2009, now issued as U.S. Pat. No. 9,063,801, both of which areincorporated in their entirety by this reference.

TECHNICAL FIELD

Embodiments are described relating to the representation, manipulation,and exchange of data within and between computing processes.

BACKGROUND

Conventional programming environments do not fully supportmulti-computer processing unit (CPU) and cross-network execution, orflexible sharing of data between large numbers of computing processes.User-facing computer programs have traditionally been constructed sothat the majority of processing and all graphical output is produced bya single computational process. This mechanism, though standard andwell-supported by tool-chains, development environments and operatingsystems, scales poorly and is a significant contributor to the bloat andbrittleness of widely-used contemporary applications.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram of a multi-process interactive system, underan embodiment.

FIG. 1B is a block diagram of a multi-process interactive system, underan alternative embodiment.

FIG. 1C is a block diagram of a multi-process interactive system, underanother alternative embodiment.

FIG. 2 is a flow diagram for operations of the multi-process interactivesystem, under an embodiment.

FIG. 3 is a block diagram of a processing environment including datarepresentations using slawx, proteins, and pools, under an embodiment.

FIG. 4 is a block diagram of a protein, under an embodiment.

FIG. 5 is a block diagram of a descrip, under an embodiment.

FIG. 6 is a block diagram of an ingest, under an embodiment.

FIG. 7 is a block diagram of a slaw, under an embodiment.

FIG. 8A is a block diagram of a protein in a pool, under an embodiment.

FIGS. 8B1 and 8B2 shows a slaw header format, under an embodiment.

FIG. 8C is a flow diagram for using proteins, under an embodiment.

FIG. 8D is a flow diagram for constructing or generating proteins, underan embodiment.

FIG. 9 is a block diagram of a processing environment including dataexchange using slawx, proteins, and pools, under an embodiment.

FIG. 10 is a block diagram of a processing environment includingmultiple devices and numerous programs running on one or more of thedevices in which the Plasma constructs (e.g., pools, proteins, and slaw)are used to allow the numerous running programs to share andcollectively respond to the events generated by the devices, under anembodiment.

FIG. 11 is a block diagram of a processing environment includingmultiple devices and numerous programs running on one or more of thedevices in which the Plasma constructs (e.g., pools, proteins, and slaw)are used to allow the numerous running programs to share andcollectively respond to the events generated by the devices, under analternative embodiment.

FIG. 12 is a block diagram of a processing environment includingmultiple input devices coupled among numerous programs running on one ormore of the devices in which the Plasma constructs (e.g., pools,proteins, and slaw) are used to allow the numerous running programs toshare and collectively respond to the events generated by the inputdevices, under another alternative embodiment.

FIG. 13 is a block diagram of a processing environment includingmultiple devices coupled among numerous programs running on one or moreof the devices in which the Plasma constructs (e.g., pools, proteins,and slaw) are used to allow the numerous running programs to share andcollectively respond to the graphics events generated by the devices,under yet another alternative embodiment.

FIG. 14 is a block diagram of a processing environment includingmultiple devices coupled among numerous programs running on one or moreof the devices in which the Plasma constructs (e.g., pools, proteins,and slaw) are used to allow stateful inspection, visualization, anddebugging of the running programs, under still another alternativeembodiment.

FIG. 15 is a block diagram of a processing environment includingmultiple devices coupled among numerous programs running on one or moreof the devices in which the Plasma constructs (e.g., pools, proteins,and slaw) are used to allow influence or control the characteristics ofstate information produced and placed in that process pool, under anadditional alternative embodiment.

FIG. 16 is a block diagram of a gestural control system, under anembodiment.

FIG. 17 is a diagram of marking tags, under an embodiment.

FIG. 18 is a diagram of poses in a gesture vocabulary, under anembodiment.

FIG. 19 is a diagram of orientation in a gesture vocabulary, under anembodiment.

FIG. 20 is a diagram of two hand combinations in a gesture vocabulary,under an embodiment.

FIG. 21 is a diagram of orientation blends in a gesture vocabulary,under an embodiment.

FIG. 22 is a flow diagram of gestural control, under an embodiment.

FIGS. 23A and 23B is an example of commands, under an embodiment.

FIG. 24 is a block diagram of a Spatial Operating Environment (SOE)implemented with a multi-process interactive system, under anembodiment.

FIG. 25 is a flow diagram for operations of the multi-processinteractive system using inputs from a gestural control system, under anembodiment.

DETAILED DESCRIPTION

Embodiments are described herein that include systems and methods forcoordinating the behaviors and outputs of multiple computer processes soas to give rise to an interactive application. The embodiments describedherein, generally referred to as a multi-process interactive system,program or application, include an application program divided into anumber of distinct computer processes capable of executing in parallel.A set of these processes is able to produce portions of the overallsystem output with which the user interacts. A set of these processeshas access to a structured and well-defined data exchange mechanism thatis used to coordinate activity. A set of these processes is operable tomake use of user input (e.g., raw user input, heavily transformed userinput, raw and heavily transformed user input, etc.) via the structureddata exchange mechanism.

The embodiments described herein provide modularity for applicationcomponents across the boundary of computational processes. As a resultof the modularity provided, the embodiments described herein providecomponent reuse, greater opportunities for interoperability, easiertesting and verification, increased robustness, and fault toleranceduring execution.

Furthermore, contemporary computers often contain multiple processorelements (CPU cores, for example). The embodiments herein scale muchbetter on multiple processor architectures than traditional applicationconstruction techniques. This “multi-core” scaling is becoming more andmore important as trends in computer design and manufacturingincreasingly favor increased core counts rather than increased clockspeeds.

The embodiments described herein enable dynamic construction,destruction and recombination of process components. The embodimentsdescribed herein enable extension of the structured data exchangemechanism across multiple computers using networking (or otherinterconnection) protocols. The embodiments described herein enabledynamic transfer of process components between computers. Theembodiments described herein enable dynamic optimization of thestructured data exchange mechanism according to the number, compositionand execution contexts of the participating processes. The embodimentsdescribed herein enable graphical output created on multiple computersto be combined together on a single display. The embodiments describedherein enable shared, coordinated graphical contexts encompassingmultiple displays. The embodiments described herein enable shared,coordinated, multi-display graphical contexts encompassing multipledisplays driven by multiple computers. The embodiments described hereinintroduce automatic history buffer built into the structured dataexchange mechanism, so that some amount of past data is always availableto application components.

The following terms are intended to have the following general meaningas they are used herein. The term “processes” as used herein meansseparable program execution contexts. Computer architectures andoperating systems differ in the technical details of processimplementation. The mechanism described here is configured to operateacross a broad range of process implementations and to facilitate hybridapplication designs or configurations that take advantage of as manyavailable computing resources as possible.

The term “device” as used herein means any processor-based devicerunning one or more programs or algorithms, any processor-based devicerunning under one or more programs or algorithms and/or any devicecoupled or connected to a processor-based device running one or moreprograms or algorithms and/or running under one or more programs oralgorithms. The term “event” as used herein means any event associatedwith a running or executing program or algorithm, a processor-baseddevice and/or a device coupled or connected to a processor-based device(e.g., an event can include, but is not limited to, an input, an output,a control, a state, a state change, an action, data (regardless offormat of the data or stage in the processing from with which the datais associated), etc.).

In the following description, numerous specific details are introducedto provide a thorough understanding of, and enabling description for,embodiments described herein. One skilled in the relevant art, however,will recognize that these embodiments can be practiced without one ormore of the specific details, or with other components, systems, etc. Inother instances, well-known structures or operations are not shown, orare not described in detail, to avoid obscuring aspects of the disclosedembodiments.

The embodiments herein include systems and methods that execute numerousprocesses on at least one processing device. The systems and methods ofan embodiment translate events of each process into data capsules. Thesystems and methods of an embodiment transfer the data capsules into anumber of pools or repositories. Each process operates as a recognizingprocess, where the recognizing process recognizes in the pools datacapsules comprising content that corresponds to an interactive functionof the recognizing process and/or an identification of the recognizingprocess. The recognizing process retrieves recognized data capsules fromthe pools and executes processing appropriate to contents of therecognized data capsules.

For example, FIG. 1A is a block diagram of a multi-process interactivesystem 10, under an embodiment. This system 10 includes a processingdevice 11 hosting or executing any number of processes P1-P7. Themulti-process interactive system 10 of this example includes or runsacross one computer 11 but is not limited to one computer and could runacross any number and/or combination of processing devices or systems.The processes P1-P7 of an embodiment include separable program executioncontexts of one or more application programs, wherein each applicationprogram comprises at least one process, but the embodiment is not solimited. The events generated or produced during execution of eachprocess are translated into some number of data capsules DC1-DC9, andthe data capsules are transferred into a number of pools 15-17 orrepositories. The oval elements of system 10 represent the pools 15-17,where the pools or repositories are a mechanism for structured dataexchange as described in detail below and in the Related Applications.The data capsules DC1-DC9, also referred to as data messages, that passthrough the pools 15-17 are generically described as “proteins,” asdescribed below.

Each process P1-P7 operates as a recognizing process, where therecognizing process recognizes in the pools 15-17 data capsulescomprising content that corresponds to an interactive function of therecognizing process P1-P7 and/or an identification of the recognizingprocess P1-P7. The recognizing process P1-P7 retrieves recognized datacapsules DC1-DC9 from the pools and executes processing appropriate tocontents of the recognized data capsules. The multi-process interactivesystem 10 in described in more detail below with reference to FIGS.3-15.

FIG. 1B is a block diagram of a multi-process interactive system 20,under an alternative embodiment. This system 20 includes a processingdevice 21 hosting or executing any number of processes P1-PX, where Xrepresents any number as appropriate to a configuration of processingdevice 21 and/or system 20. The system 20 also includes a processingdevice 22 hosting or executing any number of processes P1-PY, where Yrepresents any number as appropriate to a configuration of processingdevice 22 and/or system 20. The multi-process interactive system 20 ofthis example includes or runs across two processing devices 21/22 but isnot limited to two devices and could run across any number and/orcombination of processing devices or systems. The processes P1-PX andP1-PY of an embodiment include separable program execution contexts ofone or more application programs, wherein each application programcomprises at least one process, but the embodiment is not so limited.

The events generated or produced during execution of each process aretranslated into data capsules (not shown), and the data capsules aretransferred into one or more pools. The oval elements of system 20represent the pools, where the pools or repositories are a mechanism forstructured data exchange as described in detail below and in the RelatedApplications. In this example, a pool PL1 is hosted on processing device21, but any number of pools can be hosted on processing device 21. PoolsPL1-PLY are hosted on processing device 22, where Y represents anynumber as appropriate to a configuration of processing device 22 and/orsystem 20; any number of pools can be hosted on processing device 22.The system 20 also includes pools PL11-PLX, where X represents anynumber as appropriate to a configuration of processing device 22 and/orsystem 20; any number of pools can be hosted in system 20. Any processand/or device generating data capsules can transfer the data capsulesinto any pool in the system.

Each process P1-PX/P1-PY operates as a recognizing process, where therecognizing process recognizes in the pools data capsules comprisingcontent that corresponds to an interactive function of the recognizingprocess P1-PX/P1-PY and/or an identification of the recognizing processP1-PX/P1-PY. The recognizing process P1-PX/P1-PY retrieves recognizeddata capsules from the pools and executes processing appropriate tocontents of the recognized data capsules. The multi-process interactivesystem 20 in described in more detail below with reference to FIGS.3-15.

The embodiments herein include systems and methods that execute numerousprocesses on at least one processing device. The processes of anembodiment include separable program execution contexts of a pluralityof application programs, wherein each application program comprises atleast one process. The systems and methods of an embodiment translateevents of each process of the plurality of processes into data capsules.A data capsule includes an application-independent representation ofevent data of an event and state information of the process originatingthe data capsule. The systems and methods of an embodiment transfer thedata capsules into a number of pools or repositories. Each process of anembodiment operates as a recognizing process. The recognizing processrecognizes in the pools data capsules comprising content thatcorresponds to an interactive function of the recognizing process and/oran identification of the recognizing process. The recognizing processretrieves recognized data capsules from the pools and executesprocessing appropriate to contents of the recognized data capsules.

Examples of embodiments described herein include systems and methods forcoordinating the behaviors and graphical outputs of multiple computerprocesses to enable an interactive application. While this example isdirected to graphical processing and graphical outputs, embodiments ofthe multi-process interactive system are not limited to graphicalprocesses and can apply to any processes running under any number ofprocessing devices. The multi-process interactive system includes anapplication program divided into a number of distinct computer processescapable of executing in parallel, and a set of these processes is ableto produce portions of the overall graphical output with which a userinteracts. A set of these processes has access to a structured andwell-defined data exchange mechanism that is used to coordinate activitysuch that the set of these processes is operable to make use of userinput via the structured data exchange mechanism.

As a more specific example, the description that follows teaches amulti-process graphical program, referred to herein as Squares, as anexample instantiation of embodiments that coordinate the behaviors andgraphical outputs of multiple computer processes so as to give rise toan interactive application. The description of this exampleinstantiation is intended show how the mechanism disclosed hereinoperates at a level of detail sufficient to implement it for anyinteractive program. The mechanism (and indeed the component parts ofit) is fully general and may, in practice, be implemented in a varietyof different ways. As is typically the case in such programs, themechanism disclosed herein provides major services including, but notlimited to, access to user input, fine-grained coordination of programstate across processes, and coordination of graphical output.

The Squares program presented herein serves to demonstrate several kindsof basic coordination that are useful in real-world programs. TheSquares program provides for a flexible number of colored, translucentsquares to be rendered on one or more computer displays. Each of thesesquares is embodied in a single computational process. The states andgraphical particulars of each square depend on a variety of factors,including user input actions, the states of other squares, and globallydelivered external messages. The squares may be moved around on thedisplay using an input device (e.g., mouse, touch-screen, etc.). Agestural/spatial input system as described in the Related Applicationsmay also be used to move the squares, in which case the squares can bepositioned on any of the displays available to computers that areparticipating in the gestural/spatial network.

FIG. 1C is a block diagram of a multi-process interactive system 100,under another alternative embodiment. This system 100 includes theprocesses and interconnections that combine to form an example run ofthe program. The solid rectangle elements (e.g., elements M, P, S, G,generally) represent processes in the system 100. The oval elements(e.g., elements Ui, Coo, frames) represent pools, a mechanism forstructured data exchange as described in detail below and in the RelatedApplications. The data messages that pass through the pools aregenerically described as “proteins,” as described below.

The multi-process interactive system 100 of this example includes orruns across two computers 101 and 102 but is not limited to twocomputers and could run across any number and/or combination ofprocessing systems. In this example, a first computer 101 hostsprocesses embodying two squares S (e.g., S21, S22), and a secondcomputer 102 hosts processes embodying four squares S (e.g., S11, S12,S13, S14). Alternative embodiments can include any number of squaresprocesses S running on any number of computers. The first computer 101is coupled to a single display 110, and the second computer 102 iscoupled to three displays 121, 122, 123. Alternative embodiments caninclude any number of displays coupled to any number of computers.

Each of the two computers 101 and 102 hosts at least one ‘mouse’ processM (e.g., M1, M2). The mouse process M includes a high-level driver thattransforms computer mouse input events into a suitable stream of userinput proteins and delivers those proteins into at least one ‘userinput’ pool Ui. A gestural/spatial system (as described in detailbelow), encapsulated as a gestural/spatial process G, also delivers userinput proteins into the user input pool Ui.

Each of the two computers 101 and 102 also hosts at least one ‘pointers’process P (e.g., P1, P2). The pointers process P takes or receives datafrom the Ui pool and is responsible for determining where the user isdirecting pointer “attention”, and for drawing or rendering appropriatepointer graphics. The pointers process P places data relating to orrepresenting pointer locations and modes into a ‘coordination’ pool Coo.The pointers process P delivers graphical output into the ‘frames’ pool,which is a specialized abstraction described in detail below.

Further, each of the two computers 101 and 102 hosts several ‘square’processes S, as described above. Each square process S consults thecoordination pool Coo for pointer data and for the states of peer squareprocesses S. Each square process S also places data back into thecoordination pool Coo describing their own spatial and modal states. Thesquare process S delivers graphical output into the ‘frames’ pool. Theframes pool is a specialized abstraction, described in detail below.

The gestural/spatial process G along with the user input pool Ui andcoordination pool Coo can be hosted on either of the two computers 101and 102. Alternatively, hosting of the gestural/spatial process G alongwith the user input pool Ui and coordination pool Coo can be sharedbetween the two computers 101 and 102. As yet another alternativeconfiguration, the gestural/spatial process G along with the user inputpool Ui and coordination pool Coo can be hosted on other computers (notshown).

For proteins deposited into the local frames pool, the system 100includes a dedicated compositing process com that combines frame layersinto a single frame of output for each display, many times each second.The overall display frame-rate is generally set by a system-levelconfiguration choice, but each of the individual processes that make upthe squares application is enabled to use a different frame-rate. Thecompositing process com takes care of matching up the frame layersappropriately.

FIG. 2 is a flow diagram 200 for operations of the multi-processinteractive system, under an embodiment. The operations includeexecuting a plurality of processes on at least one processing device202. The plurality of processes include separable program executioncontexts of a plurality of application programs, such that eachapplication program comprises at least one process. The operationsinclude translating events of each process of the plurality of processesinto data capsules 204. A data message includes anapplication-independent representation of event data of an event andstate information of the process originating the data message. Theoperations include transferring the data messages into at least one poolof a plurality of pools 206. Each process operates as a recognizingprocess such that the recognizing process recognizes in the plurality ofpools data capsules comprising at least one of content that correspondsto an interactive function of the recognizing process and anidentification of the recognizing process 208. The recognizing processretrieves recognized data capsules from the plurality of pools andexecutes processing appropriate to contents of the recognized datacapsules 210. The operations of the multi-process interactive systemenable coordination among the processes, where the coordinating includeseach process of the plurality of processes coordinating with peerprocesses of the plurality of processes by retrieving from the pluralityof pools state information of the peer processes. The operations alsoenable generation of an output of the plurality of processes byinteractively combining content of a set of data capsules of at leastone pool of the plurality of pools.

In handling mouse and gestural/spatial inputs, the mouse process Mmonitors the low-level mouse hardware and translates traditional mousedriver events into proteins that are screen independent. A mouse processM protein of an embodiment delivered to the user input pool Ui, inaccordance with the description below, is as follows:

descrips:

-   -   input driver    -   mango.local    -   mouse

ingests:

-   -   id: 0x012345    -   pos: [1280,800]    -   buttons: [1]

A gestural/spatial process G protein would look similar, as follows:

descrips:

-   -   input driver    -   mango.local    -   hand

ingests:

-   -   id: 0x017845    -   gripe: . . . | | |    -   pos: [127.4, 10.5, 12.2]    -   point: [[−0.099833, 0.000000, −0.995004], [0.099335 0.995004        −0.009967]]

The pointers processes P interpret these messages as implying theposition in three-dimensional space of the various pointers that theyare responsible for drawing. A static set of pointers may have beendefined in the application code, or earlier proteins may have definedand initialized the pointers.

For the instantiation of the Squares program running within a spatialoperating environment, each of the pointers processes P knows the exactreal-world positions of the display screens attached to the computersthey are hosted by. Again, these display screens may have beeninitialized at start-up or dynamically by data messages.

As proteins arrive in the user input pool Ui, the pointers processes Preact by constructing new proteins and delivering them to thecoordination pool Coo, as follows:

descrips:

-   -   pointer    -   click    -   mouse

ingests:

-   -   mid: 0x2345    -   origin: [0.0, 0.0, 0.0]    -   passes-through: [0.0, 1625.6, −2349.3]

descrips:

-   -   pointer    -   click    -   hand    -   one-finger

ingests:

-   -   mid: 0x2372    -   origin: [127.4, 10.5, 12.2]    -   passes-through: [0.0, 1625.6, −2349.3]        These proteins or messages define the position of pointer        objects with respect to the available displays. Each pointer        process P is configured so as to manage the mathematical        transformations for the displays attached only to the computer        on which it is hosted.

Periodically, each pointer process P also draws a frame of graphicaloutput. This graphical data is delivered to the frames pool. Each frameproduced by the pointer process P renders all of the pointer graphicsthat will appear on the displays attached to the computer hosting thatprocess.

Turning to the application model and graphics of an embodiment, thesquare processes S are responsible for tracking and drawing thetranslucent squares that are the focal point of the Squares application.Each square has a position, orientation, size and color. The squareprocesses S put proteins into the coordination pool Coo any time thestate of a square changes:

descrips:

-   -   tsquare    -   position

ingests:

-   -   tid: 0x45878912    -   pos: [0.0, 0.0, 0.0]    -   up: [0.0, 1.0, 0.0]    -   over: [1.0, 0.0, 0.0]    -   size: 25.0    -   color: [1.0, 1.0, 1.0, 1.0]

The square processes S also deliver graphical output to the frames pool,much as the pointer processes do. Each square process S, however,renders its own square whether or not that square will appear on adisplay attached to the computer hosting the process. Frame handling isdescribed in detail below.

The placing or transfer of proteins describing the state of the pointersand squares into a multi-subscriber pool allows the separate processesthat make up this application to coordinate, thus providing coordinationbetween dissimilar processes. The square processes S of an embodimentmonitor for proteins that indicate a pointer passing into the area ofthe square's boundary. When this happens, the square process S puts aprotein into the pool indicating the overlap and references to thesquare and the pointer involved, as follows:

descrips:

-   -   tsquare    -   pointer-overlap    -   entrance

ingests:

-   -   tid: 0x45878912    -   mid: 0x2372

The pointers processes P monitor for proteins of this form. When apointer process P identifies or sees an overlap protein referencing it'sown mid, it changes the graphical representation it uses when drawingthe pointer frame. The overlap-indicative graphic will be used until theprocess sees an appropriate overlap-exit protein, for example:

descrips:

-   -   tsquare    -   pointer-overlap    -   exit

ingests:

-   -   tid: 0x45878912    -   mid: 0x2372

Many variations to the above coordination strategy are possible. Thepointers processes P could handle the duty of checking for geometricoverlap (rather than the squares processes S doing so as describedabove). The processes could all be frame-synchronized and an overlapprotein could be generated for every frame, which would eliminate theneed for separate entrance and exit proteins.

There is nearly always a diversity of solutions available for any givencoordination problem faced when working within the mechanism describedherein. This flexibility is, in fact, one of the strengths of theembodiments herein. The description herein documents at least one of thesolutions implemented for several of the interlocking problems faced inattempting to construct a typical multi-process graphical application. Anumber of references collect useful messaging patterns, many of whichare applicable, for example, “Enterprise Integration Patterns:Designing, Building and Deploying Messaging Solutions” by G. Hohpe andB. Woolf, ISBN 0321146530.

In connecting user input to manipulatory actions under an embodiment,interactively moving a square includes the squares processes S makinguse of the data that the pointers processes P put into the coordinationpools Coo. A squares process S initializes a move in response to thecombination of a recognized pointer overlap condition and a protein with“pointer” and “click” descrips. The graphical representation of thesquare changes while a move is in progress, and the position of thesquare in space follows the pointer. The move continues until acorresponding “pointer”/“unclick” protein occurs.

The squares of an embodiment also change their colors when they overlapone another. Whenever an S process sees a “tsquare”/“position” protein,it calculates whether there is any overlap between itself and thedepositor of that protein. If so, it uses an overlap-indicative colorwhen it renders its next frame, otherwise, it uses its normal color.

Note that the flexibility of the loosely-coupled architecture of anembodiment provides for other or alternative ways of implementing thisbehavior. The squares processes S could avoid doing the overlapcalculations and instead offload this work to another process, forexample, which would continually do the math for some or all squares anddrop proteins describing an overlap condition into the coordination poolCoo. The squares processes S would simply wait for these proteins:

descrips:

-   -   tsquare    -   square-overlap

ingests:

-   -   tids: [0x45878912, 0x45878916].

This flexibility to slice and dice the application workloads is veryuseful. Compute-intensive jobs can move to processors or machines thathave spare power. Producers of data can instantiate helper processes asneeded (and terminate them when they are no longer needed). Greatercomputational and rendering resources can be applied to applicationareas that a user is interacting directly with, or where a user isimmediately able to perceive greater granularity, detail or refreshrates.

All of this is possible because the multi-process interactive systemdescribed herein externalizes application state and allows multi-processaccess to that state. In contrast, with contemporary programming modelsruntime state is almost entirely “locked up” inside an individualprocess.

The multi-process interactive system encourages programmers to exposeall interactive functionality as protein-drivable state. The ApplicationProgramming Interface (API) is defined by the proteins that each processrecognizes, rather than by traditional function calls. For example, aprotein is defined that changes the color of any (or all) of thesquares:

descrips:

-   -   tsquare    -   color-change-command

ingests:

-   -   tids: [0x0]    -   normal-color: [1.0, 0.0, 0.0, 0.65]    -   overlap-color: [1.0, 0.0, 0.0, 1.0]

When any squares process S sees this protein in the coordination poolCoo, it checks the tids list to see whether either its own unique objectid or the general address 0x0 is present. If so, the process begins torender its square using the two new colors specified (one for the normalcase and one for the overlap case).

Using this mechanism and this “expose all interactive functions asproteins” approach, a new utility to control the color of squares can bewritten after all the other code for the Squares application is alreadyfinished, deployed and running. No recompilation or relinking isnecessary to add new functionality into the running application.

Interactive debuggers for graphical applications are another programtype that benefit from this approach. Traditional debuggers generallyneed to pause a program before they can display very much about aprograms internal state. If all of the manipulable state of a program isexposed via a pool as described herein, however, a debugger can bothmonitor and manipulate program state while the program is running.

Both the pointer processes P and square processes S push graphics datato the frames pools in order for any display output to become visible tousers. The embodiments described herein include multiple ways foroutputting graphics, some of which are described here in detail. Otherembodiments can operate under different combinations of the processesdescribed below for pushing graphics data to the frames pools andoutputting graphics.

In an embodiment, processes may use a direct rendering framework, suchas OpenGL, to draw directly to a system graphics layer. Under thisapproach, a pool is used for coordination, but not for graphics commandsor pixel data.

Another embodiment outputs graphics data via processes that deliverrendering commands to a pool. Another process (or processes) is thenresponsible for interpreting the rendering commands and driving thesystem graphics layer. These commands can be very low-level, such asbare OpenGL calls, for example. Conversely, these rendering commands canbe very high-level, for example, like the tsquare proteins describedabove that comprise sufficient information that a dedicated renderingprocess could draw the squares frame by frame.

Yet another embodiment outputs graphics data via processes that renderto in-memory pixel buffers, then transfer or place the resulting rawframe data into a pool. Another process (or processes) combines the rawframe data. The volume of data that the pool handles is generally muchlarger with this approach than for the graphics output approachesdescribed above. Local rendering and network frame transport provides agreat deal of flexibility, however, so if a high-bandwidth network and afast pools implementation is available, this is often used.

The example system 100 described above with reference to FIG. 1Cgenerally outputs graphics data via processes that render to in-memorypixel buffers, then transfers the resulting raw frame data into a pool.Another process (or processes) combines the raw frame data. The volumeof data that the pool handles is generally much larger with thisapproach than for the graphics output approaches described above. Localrendering and network frame transport provides a great deal offlexibility, however, so if a high-bandwidth network and a fast poolsimplementation is available, this is often used.

Therefore, the pointer processes P and square processes S each rendertheir own individual graphical elements. Each process chooses a numberof color components and a number of pixels to render. A process canrender a full display's worth of pixels (2560×1600, for example) usingcomponents of the RGBA (red, green, blue, alpha) color space or the RGBcolor model with alpha blending and alpha compositing. To save computecycles, rendering overhead and pools bandwidth, though, a process mayproduce only as many pixels as are necessary to capture the projectedbounding box of a particular graphical object, and may use only twocomponents if a luminance (with transparency) rendering is sufficient.

Rendered pixel data is transferred or delivered into the frames poolalong with a variety of metadata (e.g., geometric extent, layeringinformation, frame-rate indications, extra color information, etc). Asthe Squares application is running in the context of a spatial operatingenvironment, each process has access to real-world geometry data and isable to deliver appropriate output to each of the frames pools. This mayinvolve rendering more than one frame for each output cycle.

Protein deposit into a local frames pool occurs at a rate that generallymakes unnecessary compression of pixel data. To achieve relatively lowlatency for interactive applications, however, network deposits canreduce the amount of data sent for each frame. An embodiment useshardware compression to reduce the number of bytes required to representeach array of pixels, but the embodiment is not so limited.

With reference to FIG. 1C, an embodiment of the system 100 uses adedicated compositing process COM that combines these frame layers intoa single frame of output for each display, many times each second. Theoverall display frame-rate is generally set by a system-levelconfiguration choice, but each of the individual processes that make upthe squares application is enabled to use a different frame-rate. Thecompositing process com takes care of matching up the frame layersappropriately.

As described above with reference to FIGS. 1A-1C, the multi-processinteractive system of an embodiment includes processes, pools, andproteins. The solid rectangles in the system represent processes, whilethe ovals represent pools, a mechanism for structured data exchange. Thedata messages that pass through the pools are generically described as“proteins.” Each of the processes generates proteins and deposits theproteins into one or more pools, and retrieves proteins from the one ormore pools.

The pools and proteins are components of methods and systems describedherein for encapsulating data that is to be shared between or acrossprocesses. These mechanisms also include slawx (plural of “slaw”) inaddition to the proteins and pools. Generally, slawx provide thelowest-level of data definition for inter-process exchange, proteinsprovide mid-level structure and hooks for querying and filtering, andpools provide for high-level organization and access semantics. Slawxinclude a mechanism for efficient, platform-independent datarepresentation and access. Proteins provide a data encapsulation andtransport scheme using slawx as the payload. Pools provide structuredand flexible aggregation, ordering, filtering, and distribution ofproteins within a process, among local processes, across a networkbetween remote or distributed processes, and via longer term (e.g.on-disk, etc.) storage.

The configuration and implementation of embodiments of the multi-processinteractive system include several constructs that together enablenumerous capabilities. For example, the embodiments described hereinprovide efficient exchange of data between large numbers of processes asdescribed above. The embodiments described herein also provide flexibledata “typing” and structure, so that widely varying kinds and uses ofdata are supported. Furthermore, embodiments described herein includeflexible mechanisms for data exchange (e.g., local memory, disk,network, etc.), all driven by substantially similar applicationprogramming interfaces (APIs). Moreover, embodiments described enabledata exchange between processes written in different programminglanguages. Additionally, embodiments described herein enable automaticmaintenance of data caching and aggregate state.

FIG. 3 is a block diagram of a processing environment including datarepresentations using slawx, proteins, and pools, under an embodiment.The principal constructs of the embodiments presented herein includeslawx (plural of “slaw”), proteins, and pools. Slawx as described hereinincludes a mechanism for efficient, platform-independent datarepresentation and access. Proteins, as described in detail herein,provide a data encapsulation and transport scheme, and the payload of aprotein of an embodiment includes slawx. Pools, as described herein,provide structured yet flexible aggregation, ordering, filtering, anddistribution of proteins. The pools provide access to data, by virtue ofproteins, within a process, among local processes, across a networkbetween remote or distributed processes, and via ‘longer term’ (e.g.on-disk) storage.

FIG. 4 is a block diagram of a protein, under an embodiment. The proteinincludes a length header, a descrip, and an ingest. Each of the descripand ingest includes slaw or slawx, as described in detail below.

FIG. 5 is a block diagram of a descrip, under an embodiment. The descripincludes an offset, a length, and slawx, as described in detail below.

FIG. 6 is a block diagram of an ingest, under an embodiment. The ingestincludes an offset, a length, and slawx, as described in detail below.

FIG. 7 is a block diagram of a slaw, under an embodiment. The slawincludes a type header and type-specific data, as described in detailbelow.

FIG. 8A is a block diagram of a protein in a pool, under an embodiment.The protein includes a length header (“protein length”), a descripsoffset, an ingests offset, a descrip, and an ingest. The descripsincludes an offset, a length, and a slaw. The ingest includes an offset,a length, and a slaw.

The protein as described herein is a mechanism for encapsulating datathat needs to be shared between processes, or moved across a bus ornetwork or other processing structure. As an example, proteins providean improved mechanism for transport and manipulation of data includingdata corresponding to or associated with user interface events; inparticular, the user interface events of an embodiment include those ofthe gestural interface described in U.S. Pat. No. 7,598,942, and hereinincorporated by reference in its entirety. As a further example,proteins provide an improved mechanism for transport and manipulation ofdata including, but not limited to, graphics data or events, and stateinformation, to name a few. A protein is a structured record format andan associated set of methods for manipulating records. Manipulation ofrecords as used herein includes putting data into a structure, takingdata out of a structure, and querying the format and existence of data.Proteins are configured to be used via code written in a variety ofcomputer languages. Proteins are also configured to be the basicbuilding block for pools, as described herein. Furthermore, proteins areconfigured to be natively able to move between processors and acrossnetworks while maintaining intact the data they include.

In contrast to conventional data transport mechanisms, proteins areuntyped. While being untyped, the proteins provide a powerful andflexible pattern-matching facility, on top of which “type-like”functionality is implemented. Proteins configured as described hereinare also inherently multi-point (although point-to-point forms areeasily implemented as a subset of multi-point transmission).Additionally, proteins define a “universal” record format that does notdiffer (or differs only in the types of optional optimizations that areperformed) between in-memory, on-disk, and on-the-wire (network)formats, for example.

Referring to FIGS. 4 and 8, a protein of an embodiment is a linearsequence of bytes. Within these bytes are encapsulated a descrips listand a set of key-value pairs called ingests. The descrips list includesan arbitrarily elaborate but efficiently filterable per-protein eventdescription. The ingests include a set of key-value pairs that comprisethe actual contents of the protein.

Proteins' concern with key-value pairs, as well as some core ideas aboutnetwork-friendly and multi-point data interchange, is shared withearlier systems that privilege the concept of “tuples” (e.g., Linda,Jini). Proteins differ from tuple-oriented systems in several majorways, including the use of the descrips list to provide a standard,optimizable pattern matching substrate. Proteins also differ fromtuple-oriented systems in the rigorous specification of a record formatappropriate for a variety of storage and language constructs, along withseveral particular implementations of “interfaces” to that recordformat.

Turning to a description of proteins, the first four or eight bytes of aprotein specify the protein's length, which must be a multiple of 16bytes in an embodiment. This 16-byte granularity ensures thatbyte-alignment and bus-alignment efficiencies are achievable oncontemporary hardware. A protein that is not naturally “quad-wordaligned” is padded with arbitrary bytes so that its length is a multipleof 16 bytes.

The length portion of a protein has the following format: 32 bitsspecifying length, in big-endian format, with the four lowest-order bitsserving as flags to indicate macro-level protein structurecharacteristics; followed by 32 further bits if the protein's length isgreater than 2=bytes.

The 16-byte-alignment proviso of an embodiment means that the lowestorder bits of the first four bytes are available as flags. And so thefirst three low-order bit flags indicate whether the protein's lengthcan be expressed in the first four bytes or requires eight, whether theprotein uses big-endian or little-endian byte ordering, and whether theprotein employs standard or non-standard structure, respectively, butthe protein is not so limited. The fourth flag bit is reserved forfuture use.

If the eight-byte length flag bit is set, the length of the protein iscalculated by reading the next four bytes and using them as thehigh-order bytes of a big-endian, eight-byte integer (with the fourbytes already read supplying the low-order portion). If thelittle-endian flag is set, all binary numerical data in the protein isto be interpreted as little-endian (otherwise, big-endian). If thenon-standard flag bit is set, the remainder of the protein does notconform to the standard structure to be described below.

Non-standard protein structures will not be discussed further herein,except to say that there are various methods for describing andsynchronizing on non-standard protein formats available to a systemsprogrammer using proteins and pools, and that these methods can beuseful when space or compute cycles are constrained. For example, theshortest protein of an embodiment is sixteen bytes. A standard-formatprotein cannot fit any actual payload data into those sixteen bytes (thelion's share of which is already relegated to describing the location ofthe protein's component parts). But a non-standard format protein couldconceivably use 12 of its 16 bytes for data. Two applications exchangingproteins could mutually decide that any 16-byte-long proteins that theyemit always include 12 bytes representing, for example, 12 8-bit sensorvalues from a real-time analog-to-digital converter.

Immediately following the length header, in the standard structure of aprotein, two more variable-length integer numbers appear. These numbersspecify offsets to, respectively, the first element in the descrips listand the first key-value pair (ingest). These offsets are also referredto herein as the descrips offset and the ingests offset, respectively.The byte order of each quad of these numbers is specified by the proteinendianness flag bit. For each, the most significant bit of the firstfour bytes determines whether the number is four or eight bytes wide. Ifthe most significant bit (msb) is set, the first four bytes are the mostsignificant bytes of a double-word (eight byte) number. This is referredto herein as “offset form”. Use of separate offsets pointing to descripsand pairs allows descrips and pairs to be handled by different codepaths, making possible particular optimizations relating to, forexample, descrips pattern-matching and protein assembly. The presence ofthese two offsets at the beginning of a protein also allows for severaluseful optimizations.

Most proteins will not be so large as to require eight-byte lengths orpointers, so in general the length (with flags) and two offset numberswill occupy only the first three bytes of a protein. On many hardware orsystem architectures, a fetch or read of a certain number of bytesbeyond the first is “free” (e.g., 16 bytes take exactly the same numberof clock cycles to pull across the Cell processor's main bus as a singlebyte).

In many instances it is useful to allow implementation-specific orcontext-specific caching or metadata inside a protein. The use ofoffsets allows for a “hole” of arbitrary size to be created near thebeginning of the protein, into which such metadata may be slotted. Animplementation that can make use of eight bytes of metadata gets thosebytes for free on many system architectures with every fetch of thelength header for a protein.

The descrips offset specifies the number of bytes between the beginningof the protein and the first descrip entry. Each descrip entry comprisesan offset (in offset form, of course) to the next descrip entry,followed by a variable-width length field (again in offset format),followed by a slaw. If there are no further descrips, the offset is, byrule, four bytes of zeros. Otherwise, the offset specifies the number ofbytes between the beginning of this descrip entry and the next one. Thelength field specifies the length of the slaw, in bytes.

In most proteins, each descrip is a string, formatted in the slaw stringfashion: a four-byte length/type header with the most significant bitset and only the lower 30 bits used to specify length, followed by theheader's indicated number of data bytes. As usual, the length headertakes its endianness from the protein. Bytes are assumed to encode UTF-8characters (and thus—nota bene—the number of characters is notnecessarily the same as the number of bytes).

The ingests offset specifies the number of bytes between the beginningof the protein and the first ingest entry. Each ingest entry comprisesan offset (in offset form) to the next ingest entry, followed again by alength field and a slaw. The ingests offset is functionally identical tothe descrips offset, except that it points to the next ingest entryrather than to the next descrip entry.

In most proteins, every ingest is of the slaw cons type comprising atwo-value list, generally used as a key/value pair. The slaw cons recordcomprises a four-byte length/type header with the second mostsignificant bit set and only the lower 30 bits used to specify length; afour-byte offset to the start of the value (second) element; thefour-byte length of the key element; the slaw record for the keyelement; the four-byte length of the value element; and finally the slawrecord for the value element.

Generally, the cons key is a slaw string. The duplication of data acrossthe several protein and slaw cons length and offsets field provides yetmore opportunity for refinement and optimization.

The construct used under an embodiment to embed typed data insideproteins, as described above, is a tagged byte-sequence specificationand abstraction called a “slaw” (the plural is “slawx”). A slaw is alinear sequence of bytes representing a piece of (possibly aggregate)typed data, and is associated with programming-language-specific APIsthat allow slawx to be created, modified and moved around between memoryspaces, storage media, and machines. The slaw type scheme is intended tobe extensible and as lightweight as possible, and to be a commonsubstrate that can be used from any programming language.

The desire to build an efficient, large-scale inter-processcommunication mechanism is the driver of the slaw configuration.Conventional programming languages provide sophisticated data structuresand type facilities that work well in process-specific memory layouts,but these data representations invariably break down when data needs tobe moved between processes or stored on disk. The slaw architecture is,first, a substantially efficient, multi-platform friendly, low-leveldata model for inter-process communication.

But even more importantly, slawx are configured to influence, togetherwith proteins, and enable the development of future computing hardware(microprocessors, memory controllers, disk controllers). A few specificadditions to, say, the instruction sets of commonly availablemicroprocessors make it possible for slawx to become as efficient evenfor single-process, in-memory data layout as the schema used in mostprogramming languages.

Each slaw comprises a variable-length type header followed by atype-specific data layout. In an example embodiment, which supports fullslaw functionality in C, C++ and Ruby for example, types are indicatedby a universal integer defined in system header files accessible fromeach language. More sophisticated and flexible type resolutionfunctionality is also enabled: for example, indirect typing viauniversal object IDs and network lookup.

The slaw configuration of an embodiment allows slaw records to be usedas objects in language-friendly fashion from both Ruby and C++, forexample. A suite of utilities external to the C++ compiler sanity-checkslaw byte layout, create header files and macros specific to individualslaw types, and auto-generate bindings for Ruby. As a result,well-configured slaw types are quite efficient even when used fromwithin a single process. Any slaw anywhere in a process's accessiblememory can be addressed without a copy or “deserialization” step.

Slaw functionality of an embodiment includes API facilities to performone or more of the following: create a new slaw of a specific type;create or build a language-specific reference to a slaw from bytes ondisk or in memory; embed data within a slaw in type-specific fashion;query the size of a slaw; retrieve data from within a slaw; clone aslaw; and translate the endianness and other format attributes of alldata within a slaw. Every species of slaw implements the abovebehaviors.

FIG. 8B shows a slaw header format, under an embodiment. A detaileddescription of the slaw follows.

The internal structure of each slaw optimizes each of type resolution,access to encapsulated data, and size information for that slawinstance. In an embodiment, the full set of slaw types is by designminimally complete, and includes: the slaw string; the slaw cons (i.e.dyad); the slaw list; and the slaw numerical object, which itselfrepresents a broad set of individual numerical types understood aspermutations of a half-dozen or so basic attributes. The other basicproperty of any slaw is its size. In an embodiment, slawx havebyte-lengths quantized to multiples of four; these four-byte words arereferred to herein as ‘quads’. In general, such quad-based sizing alignsslawx well with the configurations of modern computer hardwarearchitectures.

The first four bytes of every slaw in an embodiment comprise a headerstructure that encodes type-description and other metainformation, andthat ascribes specific type meanings to particular bit patterns. Forexample, the first (most significant) bit of a slaw header is used tospecify whether the size (length in quad-words) of that slaw follows theinitial four-byte type header. When this bit is set, it is understoodthat the size of the slaw is explicitly recorded in the next four bytesof the slaw (e.g., bytes five through eight); if the size of the slaw issuch that it cannot be represented in four bytes (i.e. if the size is oris larger than two to the thirty-second power) then thenext-most-significant bit of the slaw's initial four bytes is also set,which means that the slaw has an eight-byte (rather than four byte)length. In that case, an inspecting process will find the slaw's lengthstored in ordinal bytes five through twelve. On the other hand, thesmall number of slaw types means that in many cases a fully specifiedtypal bit-pattern “leaves unused” many bits in the four byte slawheader; and in such cases these bits may be employed to encode theslaw's length, saving the bytes (five through eight) that wouldotherwise be required.

For example, an embodiment leaves the most significant bit of the slawheader (the “length follows” flag) unset and sets the next bit toindicate that the slaw is a “wee cons”, and in this case the length ofthe slaw (in quads) is encoded in the remaining thirty bits. Similarly,a “wee string” is marked by the pattern 001 in the header, which leavestwenty-nine bits for representation of the slaw-string's length; and aleading 0001 in the header describes a “wee list”, which by virtue ofthe twenty-eight available length-representing bits can be a slaw listof up to two-to-the-twenty-eight quads in size. A “full string” (or consor list) has a different bit signature in the header, with the mostsignificant header bit necessarily set because the slaw length isencoded separately in bytes five through eight (or twelve, in extremecases). Note that the Plasma implementation “decides” at the instant ofslaw construction whether to employ the “wee” or the “full” version ofthese constructs (the decision is based on whether the resulting sizewill “fit” in the available wee bits or not), but the full-vs.-weedetail is hidden from the user of the Plasma implementation, who knowsand cares only that she is using a slaw string, or a slaw cons, or aslaw list.

Numeric slawx are, in an embodiment, indicated by the leading headerpattern 00001. Subsequent header bits are used to represent a set oforthogonal properties that may be combined in arbitrary permutation. Anembodiment employs, but is not limited to, five such character bits toindicate whether or not the number is: (1) floating point; (2) complex;(3) unsigned; (4) “wide”; (5) “stumpy” ((4) “wide” and (5) “stumpy” arepermuted to indicate eight, sixteen, thirty-two, and sixty-four bitnumber representations). Two additional bits (e.g., (7) and (8))indicate that the encapsulated numeric data is a two-, three-, orfour-element vector (with both bits being zero suggesting that thenumeric is a “one-element vector” (i.e. a scalar)). In this embodimentthe eight bits of the fourth header byte are used to encode the size (inbytes, not quads) of the encapsulated numeric data. This size encodingis offset by one, so that it can represent any size between andincluding one and two hundred fifty-six bytes. Finally, two characterbits (e.g., (9) and (10)) are used to indicate that the numeric dataencodes an array of individual numeric entities, each of which is of thetype described by character bits (1) through (8). In the case of anarray, the individual numeric entities are not each tagged withadditional headers, but are packed as continuous data following thesingle header and, possibly, explicit slaw size information.

This embodiment affords simple and efficient slaw duplication (which canbe implemented as a byte-for-byte copy) and extremely straightforwardand efficient slaw comparison (two slawx are the same in this embodimentif and only if there is a one-to-one match of each of their componentbytes considered in sequence). This latter property is important, forexample, to an efficient implementation of the protein architecture, oneof whose critical and pervasive features is the ability to searchthrough or ‘match on’ a protein's descrips list.

Further, the embodiments herein allow aggregate slaw forms (e.g., theslaw cons and the slaw list) to be constructed simply and efficiently.For example, an embodiment builds a slaw cons from two component slawx,which may be of any type, including themselves aggregates, by: (a)querying each component slaw's size; (b) allocating memory of size equalto the sum of the sizes of the two component slawx and the one, two, orthree quads needed for the header-plus-size structure; (c) recording theslaw header (plus size information) in the first four, eight, or twelvebytes; and then (d) copying the component slawx's bytes in turn into theimmediately succeeding memory. Significantly, such a constructionroutine need know nothing about the types of the two component slawx;only their sizes (and accessibility as a sequence of bytes) matters. Thesame process pertains to the construction of slaw lists, which areordered encapsulations of arbitrarily many sub-slawx of (possibly)heterogeneous type.

A further consequence of the slaw system's fundamental format assequential bytes in memory obtains in connection with “traversal”activities—a recurring use pattern uses, for example, sequential accessto the individual slawx stored in a slaw list. The individual slawx thatrepresent the descrips and ingests within a protein structure mustsimilarly be traversed. Such maneuvers are accomplished in a stunninglystraightforward and efficient manner: to “get to” the next slaw in aslaw list, one adds the length of the current slaw to its location inmemory, and the resulting memory location is identically the header ofthe next slaw. Such simplicity is possible because the slaw and proteindesign eschews “indirection”; there are no pointers; rather, the datasimply exists, in its totality, in situ.

To the point of slaw comparison, a complete implementation of the Plasmasystem must acknowledge the existence of differing and incompatible datarepresentation schemes across and among different operating systems,CPUs, and hardware architectures. Major such differences includebyte-ordering policies (e.g., little- vs. big-endianness) andfloating-point representations; other differences exist. The Plasmaspecification requires that the data encapsulated by slawx be guaranteedinterprable (i.e., must appear in the native format of the architectureor platform from which the slaw is being inspected. This requirementmeans in turn that the Plasma system is itself responsible for dataformat conversion. However, the specification stipulates only that theconversion take place before a slaw becomes “at all visible” to anexecuting process that might inspect it. It is therefore up to theindividual implementation at which point it chooses to perform suchformat c conversion; two appropriate approaches are that slaw datapayloads are conformed to the local architecture's data format (1) as anindividual slaw is “pulled out” of a protein in which it had beenpacked, or (2) for all slaw in a protein simultaneously, as that proteinis extracted from the pool in which it was resident. Note that theconversion stipulation considers the possibility of hardware-assistedimplementations. For example, networking chipsets built with explicitPlasma capability may choose to perform format conversion intelligentlyand at the “instant of transmission”, based on the known characteristicsof the receiving system. Alternately, the process of transmission mayconvert data payloads into a canonical format, with the receivingprocess symmetrically converting from canonical to “local” format.Another embodiment performs format conversion “at the metal”, meaningthat data is always stored in canonical format, even in local memory,and that the memory controller hardware itself performs the conversionas data is retrieved from memory and placed in the registers of theproximal CPU.

A minimal (and read-only) protein implementation of an embodimentincludes operation or behavior in one or more applications orprogramming languages making use of proteins. FIG. 8B is a flow diagram850 for using proteins, under an embodiment. Operation begins byquerying 852 the length in bytes of a protein. The number of descripsentries is queried 854. The number of ingests is queried 856. A descripentry is retrieved 858 by index number. An ingest is retrieved 860 byindex number.

The embodiments described herein also define basic methods allowingproteins to be constructed and filled with data, helper-methods thatmake common tasks easier for programmers, and hooks for creatingoptimizations. FIG. 8C is a flow diagram 870 for constructing orgenerating proteins, under an embodiment. Operation begins with creation872 of a new protein. A series of descrips entries are appended 874. Aningest is also appended 876. The presence of a matching descrip isqueried 878, and the presence of a matching ingest key is queried 880.Given an ingest key, an ingest value is retrieved 882. Pattern matchingis performed 884 across descrips. Non-structured metadata is embedded886 near the beginning of the protein.

As described above, slawx provide the lowest-level of data definitionfor inter-process exchange, proteins provide mid-level structure andhooks for querying and filtering, and pools provide for high-levelorganization and access semantics. The pool is a repository forproteins, providing linear sequencing and state caching. The pool alsoprovides multi-process access by multiple programs or applications ofnumerous different types. Moreover, the pool provides a set of common,optimizable filtering and pattern-matching behaviors.

The pools of an embodiment, which can accommodate tens of thousands ofproteins, function to maintain state, so that individual processes canoffload much of the tedious bookkeeping common to multi-process programcode. A pool maintains or keeps a large buffer of past proteinsavailable—the Platonic pool is explicitly infinite—so that participatingprocesses can scan both backwards and forwards in a pool at will. Thesize of the buffer is implementation dependent, of course, but in commonusage it is often possible to keep proteins in a pool for hours or days.

The most common style of pool usage as described herein hews to abiological metaphor, in contrast to the mechanistic, point-to-pointapproach taken by existing inter-process communication frameworks. Thename protein alludes to biological inspiration: data proteins in poolsare available for flexible querying and pattern matching by a largenumber of computational processes, as chemical proteins in a livingorganism are available for pattern matching and filtering by largenumbers of cellular agents.

Two additional abstractions lean on the biological metaphor, includinguse of “handlers”, and the Golgi framework. A process that participatesin a pool generally creates a number of handlers. Handlers arerelatively small bundles of code that associate match conditions withhandle behaviors. By tying one or more handlers to a pool, a processsets up flexible call-back triggers that encapsulate state and react tonew proteins.

A process that participates in several pools generally inherits from anabstract Golgi class. The Golgi framework provides a number of usefulroutines for managing multiple pools and handlers. The Golgi class alsoencapsulates parent-child relationships, providing a mechanism for localprotein exchange that does not use a pool.

A pools API provided under an embodiment is configured to allow pools tobe implemented in a variety of ways, in order to account both forsystem-specific goals and for the available capabilities of givenhardware and network architectures. The two fundamental systemprovisions upon which pools depend are a storage facility and a means ofinter-process communication. The extant systems described herein use aflexible combination of shared memory, virtual memory, and disk for thestorage facility, and IPC queues and TCP/IP sockets for inter-processcommunication.

Pool functionality of an embodiment includes, but is not limited to, thefollowing: participating in a pool; placing a protein in a pool;retrieving the next unseen protein from a pool; rewinding orfast-forwarding through the contents (e.g., proteins) within a pool.Additionally, pool functionality can include, but is not limited to, thefollowing: setting up a streaming pool call-back for a process;selectively retrieving proteins that match particular patterns ofdescrips or ingests keys; scanning backward and forwards for proteinsthat match particular patterns of descrips or ingests keys.

The proteins described above are provided to pools as a way of sharingthe protein data contents with other applications. FIG. 9 is a blockdiagram of a processing environment including data exchange using slawx,proteins, and pools, under an embodiment. This example environmentincludes three devices (e.g., Device X, Device Y, and Device Z,collectively referred to herein as the “devices”) sharing data throughthe use of slawx, proteins and pools as described above. Each of thedevices is coupled to the three pools (e.g., Pool 1, Pool 2, Pool 3).Pool 1 includes numerous proteins (e.g., Protein X1, Protein Z2, ProteinY2, Protein X4, Protein Y4) contributed or transferred to the pool fromthe respective devices (e.g., protein Z2 is transferred or contributedto pool 1 by device Z, etc.). Pool 2 includes numerous proteins (e.g.,Protein Z4, Protein Y3, Protein Z1, Protein X3) contributed ortransferred to the pool from the respective devices (e.g., protein Y3 istransferred or contributed to pool 2 by device Y, etc.). Pool 3 includesnumerous proteins (e.g., Protein Y1, Protein Z3, Protein X2) contributedor transferred to the pool from the respective devices (e.g., protein X2is transferred or contributed to pool 3 by device X, etc.). While theexample described above includes three devices coupled or connectedamong three pools, any number of devices can be coupled or connected inany manner or combination among any number of pools, and any pool caninclude any number of proteins contributed from any number orcombination of devices. The proteins and pools of this example are asdescribed above with reference to FIGS. 3-8.

FIG. 10 is a block diagram of a processing environment includingmultiple devices and numerous programs running on one or more of thedevices in which the Plasma constructs (e.g., pools, proteins, and slaw)are used to allow the numerous running programs to share andcollectively respond to the events generated by the devices, under anembodiment. This system is but one example of a multi-user,multi-device, multi-computer interactive control scenario orconfiguration. More particularly, in this example, an interactivesystem, comprising multiple devices (e.g., device A, B, etc.) and anumber of programs (e.g., apps AA-AX, apps BA-BX, etc.) running on thedevices uses the Plasma constructs (e.g., pools, proteins, and slaw) toallow the running programs to share and collectively respond to theevents generated by these input devices.

In this example, each device (e.g., device A, B, etc.) translatesdiscrete raw data generated by or output from the programs (e.g., appsAA-AX, apps BA-BX, etc.) running on that respective device into Plasmaproteins and deposits those proteins into a Plasma pool. For example,program AX generates data or output and provides the output to device Awhich, in turn, translates the raw data into proteins (e.g., protein 1A,protein 2A, etc.) and deposits those proteins into the pool. As anotherexample, program BC generates data and provides the data to device Bwhich, in turn, translates the data into proteins (e.g., protein 1B,protein 2B, etc.) and deposits those proteins into the pool.

Each protein includes a descrip list that specifies the data or outputregistered by the application as well as identifying information for theprogram itself. Where possible, the protein descrips may also ascribe ageneral semantic meaning for the output event or action. The protein'sdata payload (e.g., ingests) carries the full set of useful stateinformation for the program event.

The proteins, as described above, are available in the pool for use byany program or device coupled or connected to the pool, regardless oftype of the program or device. Consequently, any number of programsrunning on any number of computers may extract event proteins from theinput pool. These devices need only be able to participate in the poolvia either the local memory bus or a network connection in order toextract proteins from the pool. An immediate consequence of this is thebeneficial possibility of decoupling processes that are responsible forgenerating processing events from those that use or interpret theevents. Another consequence is the multiplexing of sources and consumersof events so that devices may be controlled by one person or may be usedsimultaneously by several people (e.g., a Plasma-based input frameworksupports many concurrent users), while the resulting event streams arein turn visible to multiple event consumers.

As an example, device C can extract one or more proteins (e.g., protein1A, protein 2A, etc.) from the pool. Following protein extraction,device C can use the data of the protein, retrieved or read from theslaw of the descrips and ingests of the protein, in processing events towhich the protein data corresponds. As another example, device B canextract one or more proteins (e.g., protein 1C, protein 2A, etc.) fromthe pool. Following protein extraction, device B can use the data of theprotein in processing events to which the protein data corresponds.

Devices and/or programs coupled or connected to a pool may skimbackwards and forwards in the pool looking for particular sequences ofproteins. It is often useful, for example, to set up a program to waitfor the appearance of a protein matching a certain pattern, then skimbackwards to determine whether this protein has appeared in conjunctionwith certain others. This facility for making use of the stored eventhistory in the input pool often makes writing state management codeunnecessary, or at least significantly reduces reliance on suchundesirable coding patterns.

FIG. 11 is a block diagram of a processing environment includingmultiple devices and numerous programs running on one or more of thedevices in which the Plasma constructs (e.g., pools, proteins, and slaw)are used to allow the numerous running programs to share andcollectively respond to the events generated by the devices, under analternative embodiment. This system is but one example of a multi-user,multi-device, multi-computer interactive control scenario orconfiguration. More particularly, in this example, an interactivesystem, comprising multiple devices (e.g., devices X and Y coupled todevices A and B, respectively) and a number of programs (e.g., appsAA-AX, apps BA-BX, etc.) running on one or more computers (e.g., deviceA, device B, etc.) uses the Plasma constructs (e.g., pools, proteins,and slaw) to allow the running programs to share and collectivelyrespond to the events generated by these input devices.

In this example, each device (e.g., devices X and Y coupled to devices Aand B, respectively) is managed and/or coupled to run under or inassociation with one or more programs hosted on the respective device(e.g., device A, device B, etc.) which translates the discrete raw datagenerated by the device (e.g., device X, device A, device Y, device B,etc.) hardware into Plasma proteins and deposits those proteins into aPlasma pool. For example, device X running in association withapplication AB hosted on device A generates raw data, translates thediscrete raw data into proteins (e.g., protein 1A, protein 2A, etc.) anddeposits those proteins into the pool. As another example, device Xrunning in association with application AT hosted on device A generatesraw data, translates the discrete raw data into proteins (e.g., protein1A, protein 2A, etc.) and deposits those proteins into the pool. As yetanother example, device Z running in association with application CDhosted on device C generates raw data, translates the discrete raw datainto proteins (e.g., protein 1C, protein 2C, etc.) and deposits thoseproteins into the pool.

Each protein includes a descrip list that specifies the actionregistered by the input device as well as identifying information forthe device itself. Where possible, the protein descrips may also ascribea general semantic meaning for the device action. The protein's datapayload (e.g., ingests) carries the full set of useful state informationfor the device event.

The proteins, as described above, are available in the pool for use byany program or device coupled or connected to the pool, regardless oftype of the program or device. Consequently, any number of programsrunning on any number of computers may extract event proteins from theinput pool. These devices need only be able to participate in the poolvia either the local memory bus or a network connection in order toextract proteins from the pool. An immediate consequence of this is thebeneficial possibility of decoupling processes that are responsible forgenerating processing events from those that use or interpret theevents. Another consequence is the multiplexing of sources and consumersof events so that input devices may be controlled by one person or maybe used simultaneously by several people (e.g., a Plasma-based inputframework supports many concurrent users), while the resulting eventstreams are in turn visible to multiple event consumers.

Devices and/or programs coupled or connected to a pool may skimbackwards and forwards in the pool looking for particular sequences ofproteins. It is often useful, for example, to set up a program to waitfor the appearance of a protein matching a certain pattern, then skimbackwards to determine whether this protein has appeared in conjunctionwith certain others. This facility for making use of the stored eventhistory in the input pool often makes writing state management codeunnecessary, or at least significantly reduces reliance on suchundesirable coding patterns.

FIG. 12 is a block diagram of a processing environment includingmultiple input devices coupled among numerous programs running on one ormore of the devices in which the Plasma constructs (e.g., pools,proteins, and slaw) are used to allow the numerous running programs toshare and collectively respond to the events generated by the inputdevices, under another alternative embodiment. This system is but oneexample of a multi-user, multi-device, multi-computer interactivecontrol scenario or configuration. More particularly, in this example,an interactive system, comprising multiple input devices (e.g., inputdevices A, B, BA, and BB, etc.) and a number of programs (not shown)running on one or more computers (e.g., device A, device B, etc.) usesthe Plasma constructs (e.g., pools, proteins, and slaw) to allow therunning programs to share and collectively respond to the eventsgenerated by these input devices.

In this example, each input device (e.g., input devices A, B, BA, andBB, etc.) is managed by a software driver program hosted on therespective device (e.g., device A, device B, etc.) which translates thediscrete raw data generated by the input device hardware into Plasmaproteins and deposits those proteins into a Plasma pool. For example,input device A generates raw data and provides the raw data to device Awhich, in turn, translates the discrete raw data into proteins (e.g.,protein 1A, protein 2A, etc.) and deposits those proteins into the pool.As another example, input device BB generates raw data and provides theraw data to device B which, in turn, translates the discrete raw datainto proteins (e.g., protein 1B, protein 3B, etc.) and deposits thoseproteins into the pool.

Each protein includes a descrip list that specifies the actionregistered by the input device as well as identifying information forthe device itself. Where possible, the protein descrips may also ascribea general semantic meaning for the device action. The protein's datapayload (e.g., ingests) carries the full set of useful state informationfor the device event.

To illustrate, here are example proteins for two typical events in sucha system. Proteins are represented here as text however, in an actualimplementation, the constituent parts of these proteins are typed databundles (e.g., slaw). The protein describing a g-speak “one fingerclick” pose (described in the Related Applications) is as follows:

[Descrips: {point, engage, one, one-finger-engage, hand,

-   -   pilot-id-02, hand-id-23}

Ingests: {pilot-id=>02,

-   -   hand-id=>23,    -   pos=>[0.0, 0.0, 0.0]    -   angle-axis=>[0.0, 0.0, 0.0, 0.707]    -   gripe=>..̂∥vx    -   time=>184437103.29}]

As a further example, the protein describing a mouse click is asfollows:

[Descrips: {point, click, one, mouse-click, button-one,

-   -   mouse-id-02}

Ingests: {mouse-id=>23,

-   -   pos=>[0.0, 0.0, 0.0]    -   time=>184437124.80}]

Either or both of the sample proteins foregoing might cause aparticipating program of a host device to run a particular portion ofits code. These programs may be interested in the general semanticlabels: the most general of all, “point”, or the more specific pair,“engage, one”. Or they may be looking for events that would plausibly begenerated only by a precise device: “one-finger-engage”, or even asingle aggregate object, “hand-id-23”.

The proteins, as described above, are available in the pool for use byany program or device coupled or connected to the pool, regardless oftype of the program or device. Consequently, any number of programsrunning on any number of computers may extract event proteins from theinput pool. These devices need only be able to participate in the poolvia either the local memory bus or a network connection in order toextract proteins from the pool. An immediate consequence of this is thebeneficial possibility of decoupling processes that are responsible forgenerating ‘input events’ from those that use or interpret the events.Another consequence is the multiplexing of sources and consumers ofevents so that input devices may be controlled by one person or may beused simultaneously by several people (e.g., a Plasma-based inputframework supports many concurrent users), while the resulting eventstreams are in turn visible to multiple event consumers.

As an example or protein use, device C can extract one or more proteins(e.g., protein 1B, etc.) from the pool. Following protein extraction,device C can use the data of the protein, retrieved or read from theslaw of the descrips and ingests of the protein, in processing inputevents of input devices CA and CC to which the protein data corresponds.As another example, device A can extract one or more proteins (e.g.,protein 1B, etc.) from the pool. Following protein extraction, device Acan use the data of the protein in processing input events of inputdevice A to which the protein data corresponds.

Devices and/or programs coupled or connected to a pool may skimbackwards and forwards in the pool looking for particular sequences ofproteins. It is often useful, for example, to set up a program to waitfor the appearance of a protein matching a certain pattern, then skimbackwards to determine whether this protein has appeared in conjunctionwith certain others. This facility for making use of the stored eventhistory in the input pool often makes writing state management codeunnecessary, or at least significantly reduces reliance on suchundesirable coding patterns.

Examples of input devices that are used in the embodiments of the systemdescribed herein include gestural input sensors, keyboards, mice,infrared remote controls such as those used in consumer electronics, andtask-oriented tangible media objects, to name a few.

FIG. 13 is a block diagram of a processing environment includingmultiple devices coupled among numerous programs running on one or moreof the devices in which the Plasma constructs (e.g., pools, proteins,and slaw) are used to allow the numerous running programs to share andcollectively respond to the graphics events generated by the devices,under yet another alternative embodiment. This system is but one exampleof a system comprising multiple running programs (e.g. graphics A-E) andone or more display devices (not shown), in which the graphical outputof some or all of the programs is made available to other programs in acoordinated manner using the Plasma constructs (e.g., pools, proteins,and slaw) to allow the running programs to share and collectivelyrespond to the graphics events generated by the devices.

It is often useful for a computer program to display graphics generatedby another program. Several common examples include video conferencingapplications, network-based slideshow and demo programs, and windowmanagers. Under this configuration, the pool is used as a Plasma libraryto implement a generalized framework which encapsulates video, networkapplication sharing, and window management, and allows programmers toadd in a number of features not commonly available in current versionsof such programs.

Programs (e.g., graphics A-E) running in the Plasma compositingenvironment participate in a coordination pool through couplings and/orconnections to the pool. Each program may deposit proteins in that poolto indicate the availability of graphical sources of various kinds.Programs that are available to display graphics also deposit proteins toindicate their displays' capabilities, security and user profiles, andphysical and network locations.

Graphics data also may be transmitted through pools, or display programsmay be pointed to network resources of other kinds (RTSP streams, forexample). The phrase “graphics data” as used herein refers to a varietyof different representations that lie along a broad continuum; examplesof graphics data include but are not limited to literal examples (e.g.,an ‘image’, or block of pixels), procedural examples (e.g., a sequenceof ‘drawing’ directives, such as those that flow down a typical openGLpipeline), and descriptive examples (e.g., instructions that combineother graphical constructs by way of geometric transformation, clipping,and compositing operations).

On a local machine graphics data may be delivered throughplatform-specific display driver optimizations. Even when graphics arenot transmitted via pools, often a periodic screen-capture will bestored in the coordination pool so that clients without direct access tothe more esoteric sources may still display fall-back graphics.

The multi-process interactive system described herein, unlike mostmessage passing frameworks and network protocols, includes pools thatmaintain a significant buffer of data. So programs can rewind backwardsinto a pool looking at access and usage patterns (in the case of thecoordination pool) or extracting previous graphics frames (in the caseof graphics pools).

FIG. 14 is a block diagram of a processing environment includingmultiple devices coupled among numerous programs running on one or moreof the devices in which the Plasma constructs (e.g., pools, proteins,and slaw) are used to allow stateful inspection, visualization, anddebugging of the running programs, under still another alternativeembodiment. This system is but one example of a system comprisingmultiple running programs (e.g. program P-A, program P-B, etc.) onmultiple devices (e.g., device A, device B, etc.) in which some programsaccess the internal state of other programs using or via pools.

Most interactive computer systems comprise many programs runningalongside one another, either on a single machine or on multiplemachines and interacting across a network. Multi-program systems can bedifficult to configure, analyze and debug because run-time data ishidden inside each process and difficult to access. The generalizedframework and Plasma constructs of an embodiment described herein allowrunning programs to make much of their data available via pools so thatother programs may inspect their state. This framework enables debuggingtools that are more flexible than conventional debuggers, sophisticatedsystem maintenance tools, and visualization harnesses configured toallow human operators to analyze in detail the sequence of states that aprogram or programs has passed through.

Referring to FIG. 14, a program (e.g., program P-A, program P-B, etc.)running in this framework generates or creates a process pool uponprogram start up. This pool is registered in the system almanac, andsecurity and access controls are applied. More particularly, each device(e.g., device A, B, etc.) translates discrete raw data generated by oroutput from the programs (e.g., program P-A, program P-B, etc.) runningon that respective device into Plasma proteins and deposits thoseproteins into a Plasma pool. For example, program P-A generates data oroutput and provides the output to device A which, in turn, translatesthe raw data into proteins (e.g., protein 1A, protein 2A, protein 3A,etc.) and deposits those proteins into the pool. As another example,program P-B generates data and provides the data to device B which, inturn, translates the data into proteins (e.g., proteins 1B-4B, etc.) anddeposits those proteins into the pool.

For the duration of the program's lifetime, other programs withsufficient access permissions may attach to the pool and read theproteins that the program deposits; this represents the basic inspectionmodality, and is a conceptually “one-way” or “read-only” proposition:entities interested in a program P-A inspect the flow of statusinformation deposited by P-A in its process pool. For example, aninspection program or application running under device C can extract oneor more proteins (e.g., protein 1A, protein 2A, etc.) from the pool.Following protein extraction, device C can use the data of the protein,retrieved or read from the slaw of the descrips and ingests of theprotein, to access, interpret and inspect the internal state of programP-A.

But, recalling that the Plasma system is not only an efficient statefultransmission scheme but also an omnidirectional messaging environment,several additional modes support program-to-program state inspection. Anauthorized inspection program may itself deposit proteins into programP's process pool to influence or control the characteristics of stateinformation produced and placed in that process pool (which, after all,program P not only writes into but reads from).

FIG. 15 is a block diagram of a processing environment includingmultiple devices coupled among numerous programs running on one or moreof the devices in which the Plasma constructs (e.g., pools, proteins,and slaw) are used to allow influence or control the characteristics ofstate information produced and placed in that process pool, under anadditional alternative embodiment. In this system example, theinspection program of device C can for example request that programs(e.g., program P-A, program P-B, etc.) dump more state than normal intothe pool, either for a single instant or for a particular duration. Or,prefiguring the next ‘level’ of debug communication, an interestedprogram can request that programs (e.g., program P-A, program P-B, etc.)emit a protein listing the objects extant in its runtime environmentthat are individually capable of and available for interaction via thedebug pool. Thus informed, the interested program can ‘address’individuals among the objects in the programs runtime, placing proteinsin the process pool that a particular object alone will take up andrespond to. The interested program might, for example, request that anobject emit a report protein describing the instantaneous values of allits component variables. Even more significantly, the interested programcan, via other proteins, direct an object to change its behavior or itsvariables' values.

More specifically, in this example, inspection application of device Cplaces into the pool a request (in the form of a protein) for an objectlist (e.g., “Request-Object List”) that is then extracted by each device(e.g., device A, device B, etc.) coupled to the pool. In response to therequest, each device (e.g., device A, device B, etc.) places into thepool a protein (e.g., protein 1A, protein 1B, etc.) listing the objectsextant in its runtime environment that are individually capable of andavailable for interaction via the debug pool.

Thus informed via the listing from the devices, and in response to thelisting of the objects, the inspection application of device C addressesindividuals among the objects in the programs runtime, placing proteinsin the process pool that a particular object alone will take up andrespond to. The inspection application of device C can, for example,place a request protein (e.g., protein “Request Report P-A-O”, “RequestReport P-B-O”) in the pool that an object (e.g., object P-A-O, objectP-B-O, respectively) emit a report protein (e.g., protein 2A, protein2B, etc.) describing the instantaneous values of all its componentvariables. Each object (e.g., object P-A-O, object P-B-O) extracts itsrequest (e.g., protein “Request Report P-A-O”, “Request Report P-B-O”,respectively) and, in response, places a protein into the pool thatincludes the requested report (e.g., protein 2A, protein 2B,respectively). Device C then extracts the various report proteins (e.g.,protein 2A, protein 2B, etc.) and takes subsequent processing action asappropriate to the contents of the reports.

In this way, use of Plasma as an interchange medium tends ultimately toerode the distinction between debugging, process control, andprogram-to-program communication and coordination.

To that last, the generalized Plasma framework allows visualization andanalysis programs to be designed in a loosely-coupled fashion. Avisualization tool that displays memory access patterns, for example,might be used in conjunction with any program that outputs its basicmemory reads and writes to a pool. The programs undergoing analysis neednot know of the existence or design of the visualization tool, and viceversa.

The use of pools in the manners described above does not unduly affectsystem performance. For example, embodiments have allowed for depositingof several hundred thousand proteins per second in a pool, so thatenabling even relatively verbose data output does not noticeably inhibitthe responsiveness or interactive character of most programs.

Spatial Operating Environment (SOE)

The multi-process interactive system can be a component of or be coupledfor use with a Spatial Operating Environment (SOE). The SOE, whichincludes a gestural control system, or gesture-based control system, canalso be referred to as a Spatial User Interface (SUI) or a SpatialInterface (SI). As an example, FIG. 16 is a block diagram of a SpatialOperating Environment (SOE), under an embodiment. A user locates hishands 1601 and 1602 in the viewing area 1650 of an array of cameras1604A-1604D. The cameras detect location, orientation, and movement ofthe fingers and hands 1601 and 1602 and generate output signals topre-processor 1605. Pre-processor 1605 translates the camera output intoa gesture signal that is provided to the computer processing unit 1607of the system. The computer 1607 uses the input information to generatea command to control one or more on screen cursors and provides videooutput to display 1603.

Although the system is shown with a single user's hands as input, theSOE may be implemented using multiple users. In addition, instead of orin addition to hands, the system may track any part or parts of a user'sbody, including head, feet, legs, arms, elbows, knees, and the like.

In the embodiment shown, four cameras or sensors are used to detect thelocation, orientation, and movement of the user's hands 1601 and 1602 inthe viewing area 1650. It should be understood that the SOE may includemore (e.g., six cameras, eight cameras, etc.) or fewer (e.g., twocameras) cameras or sensors without departing from the scope or spiritof the SOE. In addition, although the cameras or sensors are disposedsymmetrically in the example embodiment, there is no requirement of suchsymmetry in the SOE. Any number or positioning of cameras or sensorsthat permits the location, orientation, and movement of the user's handsmay be used in the SOE.

In one embodiment, the cameras used are motion capture cameras capableof capturing grey-scale images. In one embodiment, the cameras used arethose manufactured by Vicon, such as the Vicon MX40 camera. This cameraincludes on-camera processing and is capable of image capture at 1000frames per second. A motion capture camera is capable of detecting andlocating markers.

In the embodiment described, the cameras are sensors used for opticaldetection. In other embodiments, the cameras or other detectors may beused for electromagnetic, magnetostatic, RFID, or any other suitabletype of detection.

Pre-processor 1605 generates three dimensional space pointreconstruction and skeletal point labeling. The gesture translator 1606converts the 3D spatial information and marker motion information into acommand language that can be interpreted by a computer processor toupdate the location, shape, and action of a cursor on a display. In analternate embodiment of the SOE, the pre-processor 1605 and gesturetranslator 1606 are integrated or combined into a single device.

Computer 1607 may be any general purpose computer such as manufacturedby Apple, Dell, or any other suitable manufacturer. The computer 1607runs applications and provides display output. Cursor information thatwould otherwise come from a mouse or other prior art input device nowcomes from the gesture system.

The SOE or an embodiment contemplates the use of marker tags on one ormore fingers of the user so that the system can locate the hands of theuser, identify whether it is viewing a left or right hand, and whichfingers are visible. This permits the system to detect the location,orientation, and movement of the users hands. This information allows anumber of gestures to be recognized by the system and used as commandsby the user.

The marker tags in one embodiment are physical tags comprising asubstrate (appropriate in the present embodiment for affixing to variouslocations on a human hand) and discrete markers arranged on thesubstrate's surface in unique identifying patterns.

The markers and the associated external sensing system may operate inany domain (optical, electromagnetic, magnetostatic, etc.) that allowsthe accurate, precise, and rapid and continuous acquisition of theirthree-space position. The markers themselves may operate either actively(e.g. by emitting structured electromagnetic pulses) or passively (e.g.by being optically retroreflective, as in the present embodiment).

At each frame of acquisition, the detection system receives theaggregate ‘cloud’ of recovered three-space locations comprising allmarkers from tags presently in the instrumented workspace volume (withinthe visible range of the cameras or other detectors). The markers oneach tag are of sufficient multiplicity and are arranged in uniquepatterns such that the detection system can perform the following tasks:(1) segmentation, in which each recovered marker position is assigned toone and only one subcollection of points that form a single tag; (2)labelling, in which each segmented subcollection of points is identifiedas a particular tag; (3) location, in which the three-space position ofthe identified tag is recovered; and (4) orientation, in which thethree-space orientation of the identified tag is recovered. Tasks (1)and (2) are made possible through the specific nature of themarker-patterns, as described below and as illustrated in one embodimentin FIG. 17.

The markers on the tags in one embodiment are affixed at a subset ofregular grid locations. This underlying grid may, as in the presentembodiment, be of the traditional Cartesian sort; or may instead be someother regular plane tessellation (a triangular/hexagonal tilingarrangement, for example). The scale and spacing of the grid isestablished with respect to the known spatial resolution of themarker-sensing system, so that adjacent grid locations are not likely tobe confused. Selection of marker patterns for all tags should satisfythe following constraint: no tag's pattern shall coincide with that ofany other tag's pattern through any combination of rotation,translation, or mirroring. The multiplicity and arrangement of markersmay further be chosen so that loss (or occlusion) of some specifiednumber of component markers is tolerated: After any arbitrarytransformation, it should still be unlikely to confuse the compromisedmodule with any other.

Referring now to FIG. 17, a number of tags 1701A-1701E (left hand) and1702A-1702E (right hand) are shown. Each tag is rectangular andcomprises in this embodiment of a 5×7 grid array. The rectangular shapeis chosen as an aid in determining orientation of the tag and to reducethe likelihood of mirror duplicates. In the embodiment shown, there aretags for each finger on each hand. In some embodiments, it may beadequate to use one, two, three, or four tags per hand. Each tag has aborder of a different grey-scale or color shade. Within this border is a3×5 grid array. Markers (represented by the black dots of FIG. 17) aredisposed at certain points in the grid array to provide information.

Qualifying information may be encoded in the tags' marker patternsthrough segmentation of each pattern into ‘common’ and ‘unique’subpatterns. For example, the present embodiment specifies two possible‘border patterns’, distributions of markers about a rectangularboundary. A ‘family’ of tags is thus established—the tags intended forthe left hand might thus all use the same border pattern as shown intags 1701A-1701E while those attached to the right hand's fingers couldbe assigned a different pattern as shown in tags 1702A-1702E. Thissubpattern is chosen so that in all orientations of the tags, the leftpattern can be distinguished from the right pattern. In the exampleillustrated, the left hand pattern includes a marker in each corner andon marker in a second from corner grid location. The right hand patternhas markers in only two corners and two markers in non corner gridlocations. An inspection of the pattern reveals that as long as anythree of the four markers are visible, the left hand pattern can bepositively distinguished from the left hand pattern. In one embodiment,the color or shade of the border can also be used as an indicator ofhandedness.

Each tag must of course still employ a unique interior pattern, themarkers distributed within its family's common border. In the embodimentshown, it has been found that two markers in the interior grid array aresufficient to uniquely identify each of the ten fingers with noduplication due to rotation or orientation of the fingers. Even if oneof the markers is occluded, the combination of the pattern and thehandedness of the tag yields a unique identifier.

In the present embodiment, the grid locations are visually present onthe rigid substrate as an aid to the (manual) task of affixing eachretroreflective marker at its intended location. These grids and theintended marker locations are literally printed via color inkjet printeronto the substrate, which here is a sheet of (initially) flexible‘shrink-film’. Each module is cut from the sheet and then oven-baked,during which thermal treatment each module undergoes a precise andrepeatable shrinkage. For a brief interval following this procedure, thecooling tag may be shaped slightly—to follow the longitudinal curve of afinger, for example; thereafter, the substrate is suitably rigid, andmarkers may be affixed at the indicated grid points.

In one embodiment, the markers themselves are three dimensional, such assmall reflective spheres affixed to the substrate via adhesive or someother appropriate means. The three-dimensionality of the markers can bean aid in detection and location over two dimensional markers. Howevereither can be used without departing from the spirit and scope of theSOE described herein.

At present, tags are affixed via Velcro or other appropriate means to aglove worn by the operator or are alternately affixed directly to theoperator's fingers using a mild double-stick tape. In a thirdembodiment, it is possible to dispense altogether with the rigidsubstrate and affix—or ‘paint’—individual markers directly onto theoperator's fingers and hands.

The SOE of an embodiment contemplates a gesture vocabulary comprisinghand poses, orientation, hand combinations, and orientation blends. Anotation language is also implemented for designing and communicatingposes and gestures in the gesture vocabulary of the SOE. The gesturevocabulary is a system for representing instantaneous ‘pose states’ ofkinematic linkages in compact textual form. The linkages in question maybe biological (a human hand, for example; or an entire human body; or agrasshopper leg; or the articulated spine of a lemur) or may instead benonbiological (e.g. a robotic arm). In any case, the linkage may besimple (the spine) or branching (the hand). The gesture vocabularysystem of the SOE establishes for any specific linkage a constant lengthstring; the aggregate of the specific ASCII characters occupying thestring's ‘character locations' is then a unique description of theinstantaneous state, or pose’, of the linkage.

FIG. 18 illustrates hand poses in an embodiment of a gesture vocabularyof the SOE, under an embodiment. The SOE supposes that each of the fivefingers on a hand is used. These fingers are codes as p-pinkie, r-ringfinger, m-middle finger, i-index finger, and t-thumb. A number of posesfor the fingers and thumbs are defined and illustrated in FIG. 18. Agesture vocabulary string establishes a single character position foreach expressible degree of freedom in the of the linkage (in this case,a finger). Further, each such degree of freedom is understood to bediscretized (or ‘quantized’), so that its full range of motion can beexpressed through assignment of one of a finite number of standard ASCIIcharacters at that string position. These degrees of freedom areexpressed with respect to a body-specific origin and coordinate system(the back of the hand, the center of the grasshopper's body; the base ofthe robotic arm; etc.). A small number of additional gesture vocabularycharacter positions are therefore used to express the position andorientation of the linkage ‘as a whole’ in the more global coordinatesystem.

Still referring to FIG. 18, a number of poses are defined and identifiedusing ASCII characters. Some of the poses are divided between thumb andnon-thumb. The SOE in this embodiment uses a coding such that the ASCIIcharacter itself is suggestive of the pose. However, any character mayused to represent a pose, whether suggestive or not. In addition, thereis no requirement in the invention to use ASCII characters for thenotation strings. Any suitable symbol, numeral, or other representationmaybe used without departing from the scope and spirit of the invention.For example, the notation may use two bits per finger if desired or someother number of bits as desired.

A curled finger is represented by the character “̂” while a curled thumbby “>”. A straight finger or thumb pointing up is indicated by “1” andat an angle by “\” or “/”. “−” represents a thumb pointing straightsideways and “x” represents a thumb pointing into the plane.

Using these individual finger and thumb descriptions, a robust number ofhand poses can be defined and written using the scheme of the invention.Each pose is represented by five characters with the order beingp-r-m-i-t as described above. FIG. 18 illustrates a number of poses anda few are described here by way of illustration and example. The handheld flat and parallel to the ground is represented by “11111”. A fistis represented by “̂̂̂̂>”. An “OK” sign is represented by “111̂>”.

The character strings provide the opportunity for straightforward ‘humanreadability’ when using suggestive characters. The set of possiblecharacters that describe each degree of freedom may generally be chosenwith an eye to quick recognition and evident analogy. For example, avertical bar (‘|’) would likely mean that a linkage element is‘straight’, an ell (‘L’) might mean a ninety-degree bend, and acircumflex (‘̂’) could indicate a sharp bend. As noted above, anycharacters or coding may be used as desired.

Any system employing gesture vocabulary strings such as described hereinenjoys the benefit of the high computational efficiency of stringcomparison—identification of or search for any specified pose literallybecomes a ‘string compare’ (e.g. UNIX's ‘strcmp( )’ function) betweenthe desired pose string and the instantaneous actual string.Furthermore, the use of ‘wildcard characters’ provides the programmer orsystem designer with additional familiar efficiency and efficacy:degrees of freedom whose instantaneous state is irrelevant for a matchmay be specified as an interrogation point (‘?’); additional wildcardmeanings may be assigned.

In addition to the pose of the fingers and thumb, the orientation of thehand can represent information. Characters describing global-spaceorientations can also be chosen transparently: the characters ‘<’, ‘>’,‘̂’, and ‘v’ may be used to indicate, when encountered in an orientationcharacter position, the ideas of left, right, up, and down. FIG. 19illustrates hand orientation descriptors and examples of coding thatcombines pose and orientation. In an embodiment, two character positionsspecify first the direction of the palm and then the direction of thefingers (if they were straight, irrespective of the fingers' actualbends). The possible characters for these two positions express a‘body-centric’ notion of orientation: ‘−’, ‘+’, ‘x’, ‘*’, ‘̂’, and ‘v’describe medial, lateral, anterior (forward, away from body), posterior(backward, away from body), cranial (upward), and caudal (downward).

In the notation scheme of and embodiment of the invention, the fivefinger pose indicating characters are followed by a colon and then twoorientation characters to define a complete command pose. In oneembodiment, a start position is referred to as an “xyz” pose where thethumb is pointing straight up, the index finger is pointing forward andthe middle finger is perpendicular to the index finger, pointing to theleft when the pose is made with the right hand. This is represented bythe string “̂̂x1−:−x”.

The ‘XYZ-hand’ is a technique for exploiting the geometry of the humanhand to allow full six-degree-of-freedom navigation of a visuallypresented three-dimensional structure. Although the technique dependsonly on the bulk translation and rotation of the operator's hand—so thatits fingers may in principal be held in any pose desired—the presentembodiment prefers a static configuration in which the index fingerpoints away from the body; the thumb points toward the ceiling; and themiddle finger points left-right. The three fingers thus describe(roughly, but with clearly evident intent) the three mutually orthogonalaxes of a three-space coordinate system: thus ‘XYZ-hand’.

XYZ-hand navigation then proceeds with the hand, fingers in a pose asdescribed above, held before the operator's body at a predetermined‘neutral location’. Access to the three translational and threerotational degrees of freedom of a three-space object (or camera) iseffected in the following natural way: left-right movement of the hand(with respect to the body's natural coordinate system) results inmovement along the computational context's x-axis; up-down movement ofthe hand results in movement along the controlled context's y-axis; andforward-back hand movement (toward/away from the operator's body)results in z-axis motion within the context. Similarly, rotation of theoperator's hand about the index finger leads to a ‘roll’ change of thecomputational context's orientation; ‘pitch’ and ‘yaw’ changes areeffected analogously, through rotation of the operator's hand about themiddle finger and thumb, respectively.

Note that while ‘computational context’ is used here to refer to theentity being controlled by the XYZ-hand method—and seems to suggesteither a synthetic three-space object or camera—it should be understoodthat the technique is equally useful for controlling the various degreesof freedom of real-world objects: the pan/tilt/roll controls of a videoor motion picture camera equipped with appropriate rotational actuators,for example. Further, the physical degrees of freedom afforded by theXYZ-hand posture may be somewhat less literally mapped even in a virtualdomain: In the present embodiment, the XYZ-hand is also used to providenavigational access to large panoramic display images, so thatleft-right and up-down motions of the operator's hand lead to theexpected left-right or up-down ‘panning’ about the image, butforward-back motion of the operator's hand maps to ‘zooming’ control.

In every case, coupling between the motion of the hand and the inducedcomputational translation/rotation may be either direct (i.e. apositional or rotational offset of the operator's hand maps one-to-one,via some linear or nonlinear function, to a positional or rotationaloffset of the object or camera in the computational context) or indirect(i.e. positional or rotational offset of the operator's hand mapsone-to-one, via some linear or nonlinear function, to a first orhigher-degree derivative of position/orientation in the computationalcontext; ongoing integration then effects a non-static change in thecomputational context's actual zero-order position/orientation). Thislatter means of control is analogous to use of a an automobile's ‘gaspedal’, in which a constant offset of the pedal leads, more or less, toa constant vehicle speed.

The ‘neutral location’ that serves as the real-world XYZ-hand's localsix-degree-of-freedom coordinate origin may be established (1) as anabsolute position and orientation in space (relative, say, to theenclosing room); (2) as a fixed position and orientation relative to theoperator herself (e.g. eight inches in front of the body, ten inchesbelow the chin, and laterally in line with the shoulder plane),irrespective of the overall position and ‘heading’ of the operator; or(3) interactively, through deliberate secondary action of the operator(using, for example, a gestural command enacted by the operator's‘other’ hand, said command indicating that the XYZ-hand's presentposition and orientation should henceforth be used as the translationaland rotational origin).

It is further convenient to provide a ‘detent’ region (or ‘dead zone’)about the XYZ-hand's neutral location, such that movements within thisvolume do not map to movements in the controlled context.

Other poses may included:

[| | | | |:vx] is a flat hand (thumb parallel to fingers) with palmfacing down and fingers forward.

[| | | | |:x̂] is a flat hand with palm facing forward and fingers towardceiling.

[| | | | |:−x] is a flat hand with palm facing toward the center of thebody (right if left hand, left if right hand) and fingers forward.

[̂̂̂−:−x] is a single-hand thumbs-up (with thumb pointing toward ceiling).

[̂̂̂|−:−x] is a mime gun pointing forward.

The SOE of an embodiment contemplates single hand commands and poses, aswell as two-handed commands and poses. FIG. 20 illustrates examples oftwo hand combinations and associated notation in an embodiment of theSOE. Reviewing the notation of the first example, “full stop” revealsthat it comprises two closed fists. The “snapshot” example has the thumband index finger of each hand extended, thumbs pointing toward eachother, defining a goal post shaped frame. The “rudder and throttle startposition” is fingers and thumbs pointing up palms facing the screen.

FIG. 21 illustrates an example of an orientation blend in an embodimentof the SOE. In the example shown the blend is represented by enclosingpairs of orientation notations in parentheses after the finger posestring. For example, the first command shows finger positions of allpointing straight. The first pair of orientation commands would resultin the palms being flat toward the display and the second pair has thehands rotating to a 45 degree pitch toward the screen. Although pairs ofblends are shown in this example, any number of blends is contemplatedin the SOE.

FIG. 23 illustrates a number of possible commands that may be used withthe SOE. Although some of the discussion here has been about controllinga cursor on a display, the SOE is not limited to that activity. In fact,the SOE has great application in manipulating any and all data andportions of data on a screen, as well as the state of the display. Forexample, the commands may be used to take the place of video controlsduring play back of video media. The commands may be used to pause, fastforward, rewind, and the like. In addition, commands may be implementedto zoom in or zoom out of an image, to change the orientation of animage, to pan in any direction, and the like. The SOE may also be usedin lieu of menu commands such as open, close, save, and the like. Inother words, any commands or activity that can be imagined can beimplemented with hand gestures.

FIG. 22 is a flow diagram of operation of the SOE, under an embodiment.At 2201 the detection system detects the markers and tags. At 2202 it isdetermined if the tags and markers are detected. If not, the systemreturns to 2201. If the tags and markers are detected at 2202, thesystem proceeds to 2203. At 2203 the system identifies the hand, fingersand pose from the detected tags and markers. At 2204 the systemidentifies the orientation of the pose. At 2205 the system identifiesthe three dimensional spatial location of the hand or hands that aredetected. (Please note that any or all of 2203, 2204, and 2205 may becombined).

At 2206 the information is translated to the gesture notation describedabove. At 2207 it is determined if the pose is valid. This may beaccomplished via a simple string comparison using the generated notationstring. If the pose is not valid, the system returns to 2201. If thepose is valid, the system sends the notation and position information tothe computer at 2208. At 2209 the computer determines the appropriateaction to take in response to the gesture and updates the displayaccordingly at 2210.

In one embodiment of the SOE, operations 2201-2205 are accomplished bythe on-camera processor. In other embodiments, the processing can beaccomplished by the system computer if desired.

The system is able to “parse” and “translate” a stream of low-levelgestures recovered by an underlying system, and turn those parsed andtranslated gestures into a stream of command or event data that can beused to control a broad range of computer applications and systems.These techniques and algorithms may be embodied in a system comprisingcomputer code that provides both an engine implementing these techniquesand a platform for building computer applications that make use of theengine's capabilities.

One embodiment is focused on enabling rich gestural use of human handsin computer interfaces, but is also able to recognize gestures made byother body parts (including, but not limited to arms, torso, legs andthe head), as well as non-hand physical tools of various kinds, bothstatic and articulating, including but not limited to calipers,compasses, flexible curve approximators, and pointing devices of variousshapes. The markers and tags may be applied to items and tools that maybe carried and used by the operator as desired.

The system described here incorporates a number of innovations that makeit possible to build gestural systems that are rich in the range ofgestures that can be recognized and acted upon, while at the same timeproviding for easy integration into applications.

The gestural parsing and translation system in one embodiment comprises:

1) a compact and efficient way to specify (encode for use in computerprograms) gestures at several different levels of aggregation:

-   -   a. a single hand's “pose” (the configuration and orientation of        the parts of the hand relative to one another) a single hand's        orientation and position in three-dimensional space.    -   b. two-handed combinations, for either hand taking into account        pose, position or both.    -   c. multi-person combinations; the system can track more than two        hands, and so more than one person can cooperatively (or        competitively, in the case of game applications) control the        target system.    -   d. sequential gestures in which poses are combined in a series;        we call these “animating” gestures.    -   e. “grapheme” gestures, in which the operator traces shapes in        space.

2) a programmatic technique for registering specific gestures from eachcategory above that are relevant to a given application context.

3) algorithms for parsing the gesture stream so that registered gesturescan be identified and events encapsulating those gestures can bedelivered to relevant application contexts.

The specification system (1), with constituent elements (1a) to (1f),provides the basis for making use of the gestural parsing andtranslating capabilities of the system described here.

A single-hand “pose” is represented as a string of

i) relative orientations between the fingers and the back of the hand,

ii) quantized into a small number of discrete states.

Using relative joint orientations allows the system described here toavoid problems associated with differing hand sizes and geometries. No“operator calibration” is required with this system. In addition,specifying poses as a string or collection of relative orientationsallows more complex gesture specifications to be easily created bycombining pose representations with further filters and specifications.

Using a small number of discrete states for pose specification makes itpossible to specify poses compactly as well as to ensure accurate poserecognition using a variety of underlying tracking technologies (forexample, passive optical tracking using cameras, active optical trackingusing lighted dots and cameras, electromagnetic field tracking, etc).

Gestures in every category (1a) to (1f) may be partially (or minimally)specified, so that non-critical data is ignored. For example, a gesturein which the position of two fingers is definitive, and other fingerpositions are unimportant, may be represented by a single specificationin which the operative positions of the two relevant fingers is givenand, within the same string, “wild cards” or generic “ignore these”indicators are listed for the other fingers.

All of the innovations described here for gesture recognition, includingbut not limited to the multi-layered specification technique, use ofrelative orientations, quantization of data, and allowance for partialor minimal specification at every level, generalize beyond specificationof hand gestures to specification of gestures using other body parts and“manufactured” tools and objects.

The programmatic techniques for “registering gestures” (2), comprise adefined set of Application Programming Interface calls that allow aprogrammer to define which gestures the engine should make available toother parts of the running system.

These API routines may be used at application set-up time, creating astatic interface definition that is used throughout the lifetime of therunning application. They may also be used during the course of the run,allowing the interface characteristics to change on the fly. Thisreal-time alteration of the interface makes it possible to,

-   -   i) build complex contextual and conditional control states,    -   ii) to dynamically add hysterisis to the control environment,        and    -   iii) to create applications in which the user is able to alter        or extend the interface vocabulary of the running system itself.

Algorithms for parsing the gesture stream (3) compare gestures specifiedas in (1) and registered as in (2) against incoming low-level gesturedata. When a match for a registered gesture is recognized, event datarepresenting the matched gesture is delivered up the stack to runningapplications.

Efficient real-time matching is desired in the design of this system,and specified gestures are treated as a tree of possibilities that areprocessed as quickly as possible.

In addition, the primitive comparison operators used internally torecognize specified gestures are also exposed for the applicationsprogrammer to use, so that further comparison (flexible state inspectionin complex or compound gestures, for example) can happen even fromwithin application contexts.

Recognition “locking” semantics are an innovation of the systemdescribed here. These semantics are implied by the registration API (2)(and, to a lesser extent, embedded within the specification vocabulary(1)). Registration API calls include,

i) “entry” state notifiers and “continuation” state notifiers, and

ii) gesture priority specifiers.

If a gesture has been recognized, its “continuation” conditions takeprecedence over all “entry” conditions for gestures of the same or lowerpriorities. This distinction between entry and continuation states addssignificantly to perceived system usability.

The system described here includes algorithms for robust operation inthe face of real-world data error and uncertainty. Data from low-leveltracking systems may be incomplete (for a variety of reasons, includingocclusion of markers in optical tracking, network drop-out or processinglag, etc).

Missing data is marked by the parsing system, and interpolated intoeither “last known” or “most likely” states, depending on the amount andcontext of the missing data.

If data about a particular gesture component (for example, theorientation of a particular joint) is missing, but the “last known”state of that particular component can be analyzed as physicallypossible, the system uses this last known state in its real-timematching.

Conversely, if the last known state is analyzed as physicallyimpossible, the system falls back to a “best guess range” for thecomponent, and uses this synthetic data in its real-time matching.

The specification and parsing systems described here have been carefullydesigned to support “handedness agnosticism,” so that for multi-handgestures either hand is permitted to satisfy pose requirements.

The system of an embodiment can provide an environment in which virtualspace depicted on one or more display devices (“screens”) is treated ascoincident with the physical space inhabited by the operator oroperators of the system. An embodiment of such an environment isdescribed here. This current embodiment includes three projector-drivenscreens at fixed locations, is driven by a single desktop computer, andis controlled using the gestural vocabulary and interface systemdescribed herein. Note, however, that any number of screens aresupported by the techniques being described; that those screens may bemobile (rather than fixed); that the screens may be driven by manyindependent computers simultaneously; and that the overall system can becontrolled by any input device or technique.

The interface system described in this disclosure should have a means ofdetermining the dimensions, orientations and positions of screens inphysical space. Given this information, the system is able todynamically map the physical space in which these screens are located(and which the operators of the system inhabit) as a projection into thevirtual space of computer applications running on the system. As part ofthis automatic mapping, the system also translates the scale, angles,depth, dimensions and other spatial characteristics of the two spaces ina variety of ways, according to the needs of the applications that arehosted by the system.

This continuous translation between physical and virtual space makespossible the consistent and pervasive use of a number of interfacetechniques that are difficult to achieve on existing applicationplatforms or that must be implemented piece-meal for each applicationrunning on existing platforms. These techniques include (but are notlimited to):

-   -   1) Use of “literal pointing”—using the hands in a gestural        interface environment, or using physical pointing tools or        devices—as a pervasive and natural interface technique.    -   2) Automatic compensation for movement or repositioning of        screens.    -   3) Graphics rendering that changes depending on operator        position, for example simulating parallax shifts to enhance        depth perception.    -   4) Inclusion of physical objects in on-screen display—taking        into account real-world position, orientation, state, etc. For        example, an operator standing in front of a large, opaque        screen, could see both applications graphics and a        representation of the true position of a scale model that is        behind the screen (and is, perhaps, moving or changing        orientation).

It is important to note that literal pointing is different from theabstract pointing used in mouse-based windowing interfaces and mostother contemporary systems. In those systems, the operator must learn tomanage a translation between a virtual pointer and a physical pointingdevice, and must map between the two cognitively.

By contrast, in the systems described in this disclosure, there is nodifference between virtual and physical space (except that virtual spaceis more amenable to mathematical manipulation), either from anapplication or user perspective, so there is no cognitive translationrequired of the operator.

The closest analogy for the literal pointing provided by the embodimentdescribed here is the touch-sensitive screen (as found, for example, onmany ATM machines). A touch-sensitive screen provides a one to onemapping between the two-dimensional display space on the screen and thetwo-dimensional input space of the screen surface. In an analogousfashion, the systems described here provide a flexible mapping(possibly, but not necessarily, one to one) between a virtual spacedisplayed on one or more screens and the physical space inhabited by theoperator. Despite the usefulness of the analogy, it is worthunderstanding that the extension of this “mapping approach” to threedimensions, an arbitrarily large architectural environment, and multiplescreens is non-trivial.

In addition to the components described herein, the system may alsoimplement algorithms implementing a continuous, systems-level mapping(perhaps modified by rotation, translation, scaling or other geometricaltransformations) between the physical space of the environment and thedisplay space on each screen.

A rendering stack which takes the computational objects and the mappingand outputs a graphical representation of the virtual space.

An input events processing stack which takes event data from a controlsystem (in the current embodiment both gestural and pointing data fromthe system and mouse input) and maps spatial data from input events tocoordinates in virtual space. Translated events are then delivered torunning applications.

A “glue layer” allowing the system to host applications running acrossseveral computers on a local area network.

Considering the description above of the SOE, the SOE can be used as acomponent of and/or coupled to a multi-process interactive system, asdescribed above with reference to FIGS. 1A-1C and elsewhere herein. TheSOE of an embodiment can be encapsulated as a gestural/spatial process Gthat delivers user input proteins into the user input pool Ui, asdescribed above.

The embodiments herein include systems and methods that detect fromgesture data a gesture made by a body. The gesture data is received viaa detector. The systems and methods of an embodiment execute numerousprocesses on a processing device. The processes generate events thatinclude a set of events representing the gesture. The systems andmethods of an embodiment translate the events of each process into datacapsules. The systems and methods of an embodiment transfer the datacapsules into numerous pools or repositories. A set of processes of thenumerous processes operate as recognizing processes. The recognizingprocesses recognize in the pools data capsules comprising content thatcorresponds to the gesture. The recognizing processes retrieverecognized data capsules from the pools and generate a gesture signalfrom the recognized data capsules by compositing contents of therecognized data capsules to form the gesture signal. The gesture signalrepresents the gesture.

FIG. 24 is a block diagram of a Spatial Operating Environment (SOE) (seeFIG. 1C, element G) implemented with or as a component of amulti-process interactive system, under an embodiment. A user locateshis hands 2401 and 2402 in the viewing area 2450 of an array of cameras2404A-2404D. The cameras detect location, orientation, and movement ofthe fingers and hands 2401 and 2402 and generate output signals topre-processor 2405. Pre-processor 2405 translates the camera output intoa gesture signal that is provided to a computer processor of the system.In this embodiment, the functions of the computer processor, performedby computer 2407 described above, can be performed by a processor ofand/or coupled to the multi-process interactive system (FIG. 1C). Thegesture signal can be provided or transferred to a pool of themulti-process interactive system (Pool Ui, FIG. 1C). Consequently, themulti-process interactive system uses the gesture signal to generate acommand to control one or more components coupled to the multi-processinteractive system (e.g., display cursor, etc.).

Although the system is shown with a single user's hands as input, theSOE may be implemented using multiple users. In addition, instead of orin addition to hands, the system may track any part or parts of a user'sbody, including head, feet, legs, arms, elbows, knees, and the like.

In the embodiment shown, four cameras or sensors are used to detect thelocation, orientation, and movement of the user's hands 2401 and 2402 inthe viewing area 2450. It should be understood that the SOE may includemore (e.g., six cameras, eight cameras, etc.) or fewer (e.g., twocameras) cameras or sensors without departing from the scope or spiritof the SOE. In addition, although the cameras or sensors are disposedsymmetrically in the example embodiment, there is no requirement of suchsymmetry in the SOE. Any number or positioning of cameras or sensorsthat permits the location, orientation, and movement of the user's handsmay be used in the SOE.

In one embodiment, the cameras used are motion capture cameras capableof capturing grey-scale images. In one embodiment, the cameras used arethose manufactured by Vicon, such as the Vicon MX40 camera. This cameraincludes on-camera processing and is capable of image capture at 1000frames per second. A motion capture camera is capable of detecting andlocating markers.

In the embodiment described, the cameras are sensors used for opticaldetection. In other embodiments, the cameras or other detectors may beused for electromagnetic, magnetostatic, RFID, or any other suitabletype of detection.

Pre-processor 2405 generates three dimensional space pointreconstruction and skeletal point labeling. The gesture translator 2406converts the 3D spatial information and marker motion information into acommand language that can be interpreted by components of themulti-process interactive system that receive information from pool Ui(see FIG. 1). In an alternate embodiment of the SOE, the pre-processor2405 and gesture translator 106 are integrated or combined into a singledevice.

FIG. 25 is a flow diagram 2500 for operations of the multi-processinteractive system 100 (see FIG. 1) using inputs from a gestural controlsystem, under an embodiment. The operations include detecting fromgesture data a gesture made by a body 2502. The gesture data is receivedvia a detector. The operations include executing a plurality ofprocesses on a processing device 2504. The processes generate eventsthat include a set of events representing the gesture. The processesinclude separable program execution contexts of a spatial operatingapplication, but are not so limited. The events of each process aretranslated into data capsules 2506. A data capsule includes anapplication-independent representation of event data of an event andstate information of the process originating the data capsule, but isnot so limited. The data capsules are transferred into a plurality ofpools 2508. A set of processes of the numerous processes operate asrecognizing processes. The recognizing processes recognize in the poolsdata capsules comprising content that corresponds to the gesture 2510.The recognizing processes retrieve recognized data capsules from thepools and generate a gesture signal from the recognized data capsules bycompositing contents of the recognized data capsules to form the gesturesignal 2512. The gesture signal represents the gesture.

Embodiments described herein include a method comprising: executing aplurality of processes on at least one processing device; translatingevents of each process of the plurality of processes into data capsules;transferring the data capsules into a plurality of pools; each processoperating as a recognizing process, the recognizing process recognizingin the plurality of pools data capsules comprising at least one ofcontent that corresponds to an interactive function of the recognizingprocess and an identification of the recognizing process; and therecognizing process retrieving recognized data capsules from theplurality of pools and executing processing appropriate to contents ofthe recognized data capsules.

A data capsule of an embodiment includes an application-independentrepresentation of event data of an event and state information of theprocess originating the data message.

The method of an embodiment comprises forming an interactive applicationfrom the plurality of processes by coordinating operations of each ofthe plurality of processes using the data capsules and the plurality ofpools.

The method of an embodiment comprises coordinating operations of theplurality of processes using at least one of the data capsules and theplurality of pools.

The method of an embodiment comprises dividing an application programinto a set of processes, wherein the plurality of processes includes theset of processes.

The method of an embodiment comprises a process generating an output byinteractively processing a plurality of retrieved data capsules of atleast one pool of the plurality of pools.

The plurality of processes of an embodiment includes separable programexecution contexts of a plurality of application programs, wherein eachapplication program comprises at least one process.

The method of an embodiment comprises executing the plurality ofprocesses in parallel.

The method of an embodiment comprises executing a first set of processesin parallel, and executing a second set of processes in sequence,wherein the plurality of processes includes the first set of processesand the second set of processes.

The events of an embodiment represent process inputs.

The events of an embodiment represent process outputs.

The events of an embodiment comprise a user interface event.

The events of an embodiment comprise a graphics event.

The events of an embodiment represent process state.

The state of a process of an embodiment represents the interactivefunction of the process, wherein the interactive function of the processis exposed to the plurality of processes as contents of the datacapsules.

The method of an embodiment comprises defining an applicationprogramming interface (API) of the plurality of processes by contents ofthe data capsules instead of defining the API by function calls.

The contents of the data capsules of an embodiment areapplication-independent and recognizable by the plurality of processes.

The at least one processing device of an embodiment comprises aplurality of processing devices.

At least one first set of processes of the plurality of processes of anembodiment are running under at least one first set of processingdevices of the plurality of processing device and at least one secondset of processes of the plurality of processes of an embodiment arerunning under at least one second set of processing devices of theplurality of processing devices.

The plurality of processes of an embodiment includes a first process.

The translating of an embodiment comprises transforming events of thefirst process into at least one data sequence comprising first processevent data specifying the event and state information of the event.

The first process event data and state information of an embodiment aretype-specific data having a type corresponding to an application of thefirst process.

The translating of an embodiment comprises forming the data capsule toinclude the at least one data sequence, the data capsule having a datastructure comprising an application-independent representation of the atleast one data sequence.

The plurality of processes of an embodiment includes a second process.

The translating of an embodiment comprises transforming state changeevents of the second process into at least one data sequence comprisingsecond process event data specifying the event and state information ofthe event.

The second process event data and state information of an embodiment aretype-specific data having a type corresponding to an application of thesecond process.

The translating of an embodiment comprises forming the data capsule toinclude the at least one data sequence, the data capsule having a datastructure comprising an application-independent representation of the atleast one data sequence.

The recognizing process of an embodiment is the second process, theretrieving comprising the second process retrieving the recognized datacapsules from the plurality of pools and executing processingappropriate to contents of the recognized data capsules.

The contents of the recognized data capsules of an embodiment are datarepresenting state information of the first process.

The translating of an embodiment comprises transforming the contents ofthe recognized data capsules into at least one new data sequence, the atleast one new data sequence representing at least one of an event of thefirst process and an event of the second process.

The at least one new data sequence of an embodiment comprises event dataspecifying the event and state information of at least one of the firstprocess and the second process.

The event data and state information of the at least one of the firstprocess and the second process of an embodiment are type-specific datahaving a type corresponding to an application of the at least one of thefirst process and the second process.

The translating of an embodiment comprises forming the data capsule toinclude the at least one new data sequence, the data capsule having adata structure comprising an application-independent representation ofthe at least one new data sequence.

The plurality of processes of an embodiment uses the at least one newdata sequence.

The processing appropriate to contents of the recognized data capsulesof an embodiment comprises rendering a graphical object, wherein thegraphical object is rendered on a display of the at least one processingdevice.

The rendering of an embodiment comprises direct rendering in which theplurality of processes draw directly to a graphics layer of the at leastone processing device, wherein the plurality of pools is used forcoordination among the plurality of processes as appropriate to therendering.

The rendering of an embodiment comprises the plurality of processestransferring data capsules comprising rendering commands into theplurality of pools. The rendering of an embodiment comprises theplurality of processes retrieving the rendering commands from theplurality of pools, interpreting the rendering commands, and driving agraphics layer of the at least one processing device in response to therendering commands.

The rendering of an embodiment comprises the plurality of processesrendering to pixel buffers. The rendering of an embodiment comprises theplurality of processes transferring raw frame data into the plurality ofpools, the raw frame data resulting from the rendering to the pixelbuffers. The rendering of an embodiment comprises the plurality ofprocesses retrieving the raw frame data from the plurality of pools andcombining the raw frame data for use in driving a graphics layer of theat least one processing device.

The method of an embodiment comprises detecting an event of theplurality of processes. The method of an embodiment comprises generatingat least one data sequence comprising event data specifying the eventand state information of the event, wherein the event data and stateinformation are type-specific data having a type corresponding to anapplication of the at least one processing device. The method of anembodiment comprises forming a data capsule to include the at least onedata sequence, the data capsule having a data structure comprising anapplication-independent representation of the at least one datasequence.

The generating of the at least one data sequence of an embodimentcomprises generating a first respective data set that includes firstrespective event data. The generating of the at least one data sequenceof an embodiment comprises generating a second respective data set thatincludes second respective state information. The generating of the atleast one data sequence of an embodiment comprises forming a first datasequence to include the first respective data set and the secondrespective data set.

The generating of the first respective data set of an embodimentincludes forming the first respective data set to include identificationdata of the at least one processing device, the identification dataincluding data identifying the at least one processing device.

The generating of the at least one data sequence of an embodimentcomprises generating a first respective data set that includes firstrespective event data. The generating of the at least one data sequenceof an embodiment comprises generating a second respective data set thatincludes second respective state information. The generating of the atleast one data sequence of an embodiment comprises forming a second datasequence to include the first respective data set and the secondrespective data set.

The generating of the first respective data set of an embodimentincludes generating a first respective data set offset, wherein thefirst respective data set offset points to the first respective data setof the second data sequence.

The generating of the second respective data set of an embodimentincludes generating a second respective data set offset, wherein thesecond respective data set offset points to the second respective dataset of the second data sequence.

The first respective data set of an embodiment is a description list,the description list including a description of the data.

The event data of an embodiment is a tagged byte-sequence representingtyped data.

The event data of an embodiment includes a type header and atype-specific data layout.

The state information of an embodiment is a tagged byte-sequencerepresenting typed data.

The state information of an embodiment includes a type header and atype-specific data layout.

The method of an embodiment comprises generating at least one offset.The method of an embodiment comprises forming the data capsule toinclude the at least one offset.

The method of an embodiment comprises generating a first offset having afirst variable length, wherein the first offset points to the event dataof a first data sequence of the at least one data sequence.

The method of an embodiment comprises generating a second offset havinga second variable length, wherein the second offset points to the stateinformation of a first data sequence of the at least one data sequence.

The method of an embodiment comprises forming a first code path throughthe data capsule using a first offset of the at least one offset. Themethod of an embodiment comprises forming a second code path through thedata capsule using a second offset of the at least one offset, whereinthe first code path and the second code path are different paths.

At least one of the first offset and the second offset of an embodimentinclude metadata, the metadata comprising context-specific metadatacorresponding to a context of the application.

The method of an embodiment comprises generating a header that includesa length of the data capsule. The method of an embodiment comprisesforming the data capsule to include the header.

The method of an embodiment comprises transferring the data capsule to apool of the plurality of pools.

The method of an embodiment comprises detecting a second event of the atleast one processing device. The method of an embodiment comprisessearching the plurality of pools for data capsules corresponding to thesecond event.

The method of an embodiment comprises identifying a correspondencebetween the data capsule and the second event. The method of anembodiment comprises extracting the data capsule from the pool inresponse to the identifying. The method of an embodiment comprisesexecuting on behalf of the at least one processing device a processingoperation corresponding to the second event in response to contents ofthe data capsule, wherein the at least one processing device correspondsto an application of a first type and a second application of a secondtype.

The plurality of pools of an embodiment is coupled to a plurality ofapplications, the plurality of pools including a plurality of datacapsules corresponding to the plurality of applications, the pluralityof pools providing access to the plurality of data capsules by theplurality of applications, wherein at least two applications of theplurality of applications are different applications.

The plurality of pools of an embodiment provides state caching of aplurality of data capsules.

The plurality of pools of an embodiment provides linear sequencing of aplurality of data capsules.

The data structure of an embodiment is untyped.

The data structure of the data capsule of an embodiment provides aplatform-independent representation of the event data and the stateinformation.

The data structure of the data capsule of an embodiment providesplatform-independent access to the event data and the state information.

The transferring of an embodiment comprises transferring the datacapsule from a first application having a first application type to atleast one second application having at least one second applicationtype, wherein the first application type is different than the secondapplication type, wherein the generating of the at least one datasequence was executed by the first application, the method comprisingmaintaining intact the at least one data sequence of the data capsuleduring the transferring.

The method of an embodiment comprises using the at least one datasequence during operations of the second application.

The method of an embodiment comprises generating a first data set thatincludes event data and identification data of a source device of the atleast one processing device, the device event data including dataspecifying an event registered by the source device, the identificationdata including data identifying the source device.

The method of an embodiment comprises generating a second data set thatincludes a full set of state information of the event, wherein each ofthe first data set and the second data set comprise typed data bundlesin a type-specific data layout.

The translating of an embodiment comprises encapsulating the first dataset and the second data set by forming a data capsule to include thefirst data set and the second data set, wherein the data capsule has adata structure comprising an application-independent representation ofthe at least one data sequence.

The method of an embodiment comprises detecting an event of a firstprocessing device running under an application of a first type. Themethod of an embodiment comprises generating data sequences comprisingevent data of the first processing device, the event data specifying theevent and state information of the event, wherein the event data andstate information are type-specific data having a type corresponding tothe application. The method of an embodiment comprises forming a datacapsule to include the data sequences, the data capsule having a datastructure comprising an application-independent representation of thedata sequences. The method of an embodiment comprises detecting a secondevent of a second processing device running under at least one secondapplication having at least one second type, wherein the second type isdifferent from the first type, wherein the at least one processingdevice comprises the first processing device and the second processingdevice. The method of an embodiment comprises identifying acorrespondence between the data capsule and the second event. The methodof an embodiment comprises executing an operation in response to thesecond event using contents of the data sequences of the data capsule.

The generating of the data sequences of an embodiment comprisesgenerating a first data set that includes the event data. The generatingof the data sequences of an embodiment comprises generating a seconddata set that includes the state information. The generating of the datasequences of an embodiment comprises forming a first data sequence toinclude the first data set and the second data set.

The event data of an embodiment is a tagged byte-sequence representingtyped data.

The event data of an embodiment includes a type header and atype-specific data layout.

The state information of an embodiment is a tagged byte-sequencerepresenting typed data.

The state information of an embodiment includes a type header and atype-specific data layout.

The method of an embodiment comprises generating at least one offset.The method of an embodiment comprises forming the data capsule toinclude the at least one offset.

The method of an embodiment comprises generating a first offset having afirst variable length, wherein the first offset points to the event dataof a first data sequence of the at least one data sequence. The methodof an embodiment comprises generating a second offset having a secondvariable length, wherein the second offset points to the stateinformation of a first data sequence of the at least one data sequence.

The method of an embodiment comprises forming a first code path throughthe data capsule using a first offset of the at least one offset. Themethod of an embodiment comprises forming a second code path through thedata capsule using a second offset of the at least one offset, whereinthe first code path and the second code path are different paths.

At least one of the first offset and the second offset of an embodimentinclude metadata, the metadata comprising context-specific metadatacorresponding to a context of the application.

The method of an embodiment comprises transferring the data capsule to apool of the plurality of pools.

The method of an embodiment comprises searching the plurality of poolsfor data capsules corresponding to the second event. The method of anembodiment comprises extracting the data capsule from the pool inresponse to the identifying of the correspondence.

The plurality of pools of an embodiment is coupled to the applicationand the at least one second application, the plurality of poolsincluding a plurality of data capsules corresponding to the applicationand the at least one second application, the plurality of poolsproviding access to the plurality of data capsules by the applicationand the at least one second application.

The plurality of pools of an embodiment provides state caching of aplurality of data capsules.

The plurality of pools of an embodiment provides linear sequencing of aplurality of data capsules.

The data structure of an embodiment is untyped.

The data structure of the data capsule of an embodiment provides aplatform-independent representation of the event data and the stateinformation.

The data structure of the data capsule of an embodiment providesplatform-independent access to the event data and the state information.

Embodiments described herein include a method comprising: executing aplurality of processes on a processing device, the plurality ofprocesses including separable program execution contexts of a pluralityof application programs, wherein each application program comprises atleast one process; translating events of each process of the pluralityof processes into data messages, wherein a data message includes anapplication-independent representation of event data of an event andstate information of the process originating the data message;transferring the data messages into at least one pool of a plurality ofpools; coordinating among the processes, the coordinating including eachprocess of the plurality of processes coordinating with peer processesof the plurality of processes by retrieving from the plurality of poolsthe state information of the peer processes; and generating an output ofthe plurality of processes by interactively combining a set of datamessages of at least one pool of the plurality of pools.

Embodiments described herein include a system comprising: at least oneprocessing device, the processing device executing a plurality ofprocesses; and a plurality of pools coupled to the at least oneprocessing device; the at least one processing device translating eventsof each process of the plurality of processes into data capsules andtransferring the data capsules into the plurality of pools; each processof the plurality of processes operating as a recognizing process, therecognizing process recognizing in the plurality of pools data capsulescomprising at least one of content that corresponds to an interactivefunction of the recognizing process and an identification of therecognizing process; the recognizing process retrieving recognized datacapsules from the plurality of pools and executing processingappropriate to contents of the recognized data capsules.

Embodiments described herein include a method comprising: executing aplurality of processes on at least one processing device, the pluralityof processes including separable program execution contexts of aplurality of application programs, wherein each application programcomprises at least one process; translating events of each process ofthe plurality of processes into data capsules, wherein a data capsuleincludes an application-independent representation of event data of anevent and state information of the process originating the data capsule;transferring the data capsules into a plurality of pools; each processoperating as a recognizing process, the recognizing process recognizingin the plurality of pools data capsules comprising at least one ofcontent that corresponds to an interactive function of the recognizingprocess and an identification of the recognizing process; and therecognizing process retrieving recognized data capsules from theplurality of pools and executing processing appropriate to contents ofthe recognized data capsules.

The method of an embodiment comprises forming an interactive applicationfrom the plurality of processes by coordinating operations of each ofthe plurality of processes using the data capsules and the plurality ofpools.

The method of an embodiment comprises coordinating operations of theplurality of processes using at least one of the data capsules and theplurality of pools.

The method of an embodiment comprises dividing an application programinto a set of processes, wherein the plurality of processes includes theset of processes.

The method of an embodiment comprises a process generating an output byinteractively processing a plurality of retrieved data capsules of atleast one pool of the plurality of pools.

The method of an embodiment comprises executing the plurality ofprocesses in parallel.

The method of an embodiment comprises executing a first set of processesin parallel, and executing a second set of processes in sequence,wherein the plurality of processes includes the first set of processesand the second set of processes.

The events of an embodiment represent process inputs.

The events of an embodiment represent process outputs.

The events of an embodiment comprise a user interface event.

The events of an embodiment comprise a graphics event.

The events of an embodiment represent process state.

The state of a process of an embodiment represents the interactivefunction of the process, wherein the interactive function of the processis exposed to the plurality of processes as contents of the datacapsules.

The method of an embodiment comprises defining an applicationprogramming interface (API) of the plurality of processes by contents ofthe data capsules instead of defining the API by function calls.

The contents of the data capsules of an embodiment areapplication-independent and recognizable by the plurality of processes.

The at least one processing device of an embodiment comprises aplurality of processing devices.

At least one first set of processes of the plurality of processes of anembodiment are running under at least one first set of processingdevices of the plurality of processing device and at least one secondset of processes of the plurality of processes of an embodiment arerunning under at least one second set of processing devices of theplurality of processing devices.

The plurality of processes of an embodiment includes a first process.

The translating of an embodiment comprises transforming events of thefirst process into at least one data sequence comprising first processevent data specifying the event and state information of the event.

The first process event data and state information of an embodiment aretype-specific data having a type corresponding to an application of thefirst process.

The translating of an embodiment comprises forming the data capsule toinclude the at least one data sequence, the data capsule having a datastructure comprising an application-independent representation of the atleast one data sequence.

The plurality of processes of an embodiment includes a second process.

The translating of an embodiment comprises transforming state changeevents of the second process into at least one data sequence comprisingsecond process event data specifying the event and state information ofthe event.

The second process event data and state information of an embodiment aretype-specific data having a type corresponding to an application of thesecond process.

The translating of an embodiment comprises forming the data capsule toinclude the at least one data sequence, the data capsule having a datastructure comprising an application-independent representation of the atleast one data sequence.

The recognizing process of an embodiment is the second process, theretrieving comprising the second process retrieving the recognized datacapsules from the plurality of pools and executing processingappropriate to contents of the recognized data capsules.

The contents of the recognized data capsules of an embodiment are datarepresenting state information of the first process.

The translating of an embodiment comprises transforming the contents ofthe recognized data capsules into at least one new data sequence, the atleast one new data sequence representing at least one of an event of thefirst process and an event of the second process.

The at least one new data sequence of an embodiment comprises event dataspecifying the event and state information of at least one of the firstprocess and the second process.

The event data and state information of the at least one of the firstprocess and the second process of an embodiment are type-specific datahaving a type corresponding to an application of the at least one of thefirst process and the second process.

The translating of an embodiment comprises forming the data capsule toinclude the at least one new data sequence, the data capsule having adata structure comprising an application-independent representation ofthe at least one new data sequence.

The plurality of processes of an embodiment uses the at least one newdata sequence.

The plurality of processes of an embodiment includes an input process,the input process receiving input events from an input device.

The translating of an embodiment comprises transforming the input eventsof the input device into at least one data sequence comprising inputdevice event data specifying the event and state information of theevent.

The input device event data and state information of an embodiment aretype-specific data having a type corresponding to an application of thesource device.

The translating of an embodiment comprises forming the data capsule toinclude the at least one data sequence, the data capsule having a datastructure comprising an application-independent representation of the atleast one data sequence.

The plurality of processes of an embodiment includes a pointer process.

The recognizing process of an embodiment is the pointer process, theretrieving comprising the pointer process retrieving the recognized datacapsules from the plurality of pools and executing processingappropriate to contents of the recognized data capsules.

The contents of the recognized data capsules of an embodiment are datarepresenting input events from an input process.

The contents of the recognized data capsules of an embodiment are datarepresenting a position on a display where a user of the at least oneprocessing device is directing a pointer object.

The translating of an embodiment comprises transforming the contents ofthe recognized data capsules into at least one new data sequence, the atleast one new data sequence defining a position of the pointer objectwith respect to a display.

The at least one new data sequence of an embodiment comprises pointerprocess event data specifying the event and state information of thepointer process event.

The pointer process event data and state information of an embodimentare type-specific data having a type corresponding to an application ofthe pointer process.

The translating of an embodiment comprises forming the data capsule toinclude the at least one new data sequence, the data capsule having adata structure comprising an application-independent representation ofthe at least one new data sequence.

The plurality of processes of an embodiment uses the at least one newdata sequence in rendering the pointer object on the display.

The plurality of processes of an embodiment includes a graphicalprocess.

The translating of an embodiment comprises transforming state changeevents of the graphical process into at least one data sequencecomprising graphical process event data specifying the event and stateinformation of the event.

The graphical process event data and state information of an embodimentare type-specific data having a type corresponding to an application ofthe graphical process.

The translating of an embodiment comprises forming the data capsule toinclude the at least one data sequence, the data capsule having a datastructure comprising an application-independent representation of the atleast one data sequence.

The recognizing process of an embodiment is the graphical process, theretrieving comprising the graphical process retrieving the recognizeddata capsules from the plurality of pools and executing processingappropriate to contents of the recognized data capsules.

The contents of the recognized data capsules of an embodiment are datarepresenting state information of another process of the plurality ofprocesses.

The state information of an embodiment includes information of at leastone of spatial state and modal state.

The contents of the recognized data capsules of an embodiment are datarepresenting a position on a display where a user of the at least oneprocessing device is directing a pointer object.

The position of the pointer object of an embodiment is within a boundaryof a graphical object, wherein the graphical object is rendered by thegraphical process.

The translating of an embodiment comprises transforming the contents ofthe recognized data capsules into at least one new data sequence, the atleast one new data sequence representing at least one of the graphicalobject, the pointer object, and an overlap of the pointer object and theboundary.

The at least one new data sequence of an embodiment comprises graphicalprocess event data specifying the event and state information of thegraphical process event.

The graphical process event data and state information of an embodimentare type-specific data having a type corresponding to an application ofthe graphical process.

The translating of an embodiment comprises forming the data capsule toinclude the at least one new data sequence, the data capsule having adata structure comprising an application-independent representation ofthe at least one new data sequence.

The plurality of processes of an embodiment uses the at least one newdata sequence in rendering at least one of the graphical object and thepointer object on the display.

The processing appropriate to contents of the recognized data capsulesof an embodiment comprises rendering a graphical object, wherein thegraphical object is rendered on a display of the at least one processingdevice.

The rendering of an embodiment comprises direct rendering in which theplurality of processes draw directly to a graphics layer of the at leastone processing device, wherein the plurality of pools is used forcoordination among the plurality of processes as appropriate to therendering.

The rendering of an embodiment comprises the plurality of processestransferring data capsules comprising rendering commands into theplurality of pools. The rendering of an embodiment comprises theplurality of processes retrieving the rendering commands from theplurality of pools, interpreting the rendering commands, and driving agraphics layer of the at least one processing device in response to therendering commands.

The rendering of an embodiment comprises the plurality of processesrendering to pixel buffers. The rendering of an embodiment comprises theplurality of processes transferring raw frame data into the plurality ofpools, the raw frame data resulting from the rendering to the pixelbuffers. The rendering of an embodiment comprises the plurality ofprocesses retrieving the raw frame data from the plurality of pools andcombining the raw frame data for use in driving a graphics layer of theat least one processing device.

The method of an embodiment comprises detecting an event of theplurality of processes. The method of an embodiment comprises generatingat least one data sequence comprising event data specifying the eventand state information of the event, wherein the event data and stateinformation are type-specific data having a type corresponding to anapplication of the at least one processing device. The method of anembodiment comprises forming a data capsule to include the at least onedata sequence, the data capsule having a data structure comprising anapplication-independent representation of the at least one datasequence.

The generating of the at least one data sequence of an embodimentcomprises generating a first respective data set that includes firstrespective event data. The generating of the at least one data sequenceof an embodiment comprises generating a second respective data set thatincludes second respective state information. The generating of the atleast one data sequence of an embodiment comprises forming a first datasequence to include the first respective data set and the secondrespective data set.

The generating of the first respective data set of an embodimentincludes forming the first respective data set to include identificationdata of the at least one processing device, the identification dataincluding data identifying the at least one processing device.

The generating of the at least one data sequence of an embodimentcomprises generating a first respective data set that includes firstrespective event data. The generating of the at least one data sequenceof an embodiment comprises generating a second respective data set thatincludes second respective state information. The generating of the atleast one data sequence of an embodiment comprises forming a second datasequence to include the first respective data set and the secondrespective data set.

The generating of the first respective data set of an embodimentincludes generating a first respective data set offset, wherein thefirst respective data set offset points to the first respective data setof the second data sequence.

The generating of the second respective data set of an embodimentincludes generating a second respective data set offset, wherein thesecond respective data set offset points to the second respective dataset of the second data sequence.

The first respective data set of an embodiment is a description list,the description list including a description of the data.

The event data of an embodiment is a tagged byte-sequence representingtyped data.

The event data of an embodiment includes a type header and atype-specific data layout.

The state information of an embodiment is a tagged byte-sequencerepresenting typed data.

The state information of an embodiment includes a type header and atype-specific data layout.

The method of an embodiment comprises generating at least one offset.The method of an embodiment comprises forming the data capsule toinclude the at least one offset.

The method of an embodiment comprises generating a first offset having afirst variable length, wherein the first offset points to the event dataof a first data sequence of the at least one data sequence.

The method of an embodiment comprises generating a second offset havinga second variable length, wherein the second offset points to the stateinformation of a first data sequence of the at least one data sequence.

The method of an embodiment comprises forming a first code path throughthe data capsule using a first offset of the at least one offset. Themethod of an embodiment comprises forming a second code path through thedata capsule using a second offset of the at least one offset, whereinthe first code path and the second code path are different paths.

At least one of the first offset and the second offset of an embodimentinclude metadata, the metadata comprising context-specific metadatacorresponding to a context of the application.

The method of an embodiment comprises generating a header that includesa length of the data capsule. The method of an embodiment comprisesforming the data capsule to include the header.

The method of an embodiment comprises transferring the data capsule to apool of the plurality of pools.

The method of an embodiment comprises detecting a second event of the atleast one processing device. The method of an embodiment comprisessearching the plurality of pools for data capsules corresponding to thesecond event.

The method of an embodiment comprises identifying a correspondencebetween the data capsule and the second event. The method of anembodiment comprises extracting the data capsule from the pool inresponse to the identifying. The method of an embodiment comprisesexecuting on behalf of the at least one processing device a processingoperation corresponding to the second event in response to contents ofthe data capsule, wherein the at least one processing device correspondsto an application of a first type and a second application of a secondtype.

The plurality of pools of an embodiment is coupled to a plurality ofapplications, the plurality of pools including a plurality of datacapsules corresponding to the plurality of applications, the pluralityof pools providing access to the plurality of data capsules by theplurality of applications, wherein at least two applications of theplurality of applications are different applications.

The plurality of pools of an embodiment provides state caching of aplurality of data capsules.

The plurality of pools of an embodiment provides linear sequencing of aplurality of data capsules.

The data structure of an embodiment is untyped.

The data structure of the data capsule of an embodiment provides aplatform-independent representation of the event data and the stateinformation.

The data structure of the data capsule of an embodiment providesplatform-independent access to the event data and the state information.

The transferring comprises transferring the data capsule from a firstapplication having a first application type to at least one secondapplication having at least one second application type, wherein thefirst application type is different than the second application type,wherein the generating of the at least one data sequence was executed bythe first application, the method comprising maintaining intact the atleast one data sequence of the data capsule during the transferring.

The method of an embodiment comprises using the at least one datasequence during operations of the second application.

The method of an embodiment comprises generating a first data set thatincludes event data and identification data of a source device of the atleast one processing device, the device event data including dataspecifying an event registered by the source device, the identificationdata including data identifying the source device.

The method of an embodiment comprises generating a second data set thatincludes a full set of state information of the event, wherein each ofthe first data set and the second data set comprise typed data bundlesin a type-specific data layout.

The translating of an embodiment comprises encapsulating the first dataset and the second data set by forming a data capsule to include thefirst data set and the second data set, wherein the data capsule has adata structure comprising an application-independent representation ofthe at least one data sequence.

The method of an embodiment comprises: detecting an event of a firstprocessing device running under an application of a first type;generating data sequences comprising event data of the first processingdevice, the event data specifying the event and state information of theevent, wherein the event data and state information are type-specificdata having a type corresponding to the application; forming a datacapsule to include the data sequences, the data capsule having a datastructure comprising an application-independent representation of thedata sequences; detecting a second event of a second processing devicerunning under at least one second application having at least one secondtype, wherein the second type is different from the first type, whereinthe at least one processing device comprises the first processing deviceand the second processing device; identifying a correspondence betweenthe data capsule and the second event; and executing an operation inresponse to the second event using contents of the data sequences of thedata capsule.

The generating of the data sequences of an embodiment comprisesgenerating a first data set that includes the event data. The generatingof the data sequences of an embodiment comprises generating a seconddata set that includes the state information. The generating of the datasequences of an embodiment comprises forming a first data sequence toinclude the first data set and the second data set.

The event data of an embodiment is a tagged byte-sequence representingtyped data.

The event data of an embodiment includes a type header and atype-specific data layout.

The state information of an embodiment is a tagged byte-sequencerepresenting typed data.

The state information of an embodiment includes a type header and atype-specific data layout.

The method of an embodiment comprises generating at least one offset.The method of an embodiment comprises forming the data capsule toinclude the at least one offset.

The method of an embodiment comprises generating a first offset having afirst variable length, wherein the first offset points to the event dataof a first data sequence of the at least one data sequence. The methodof an embodiment comprises generating a second offset having a secondvariable length, wherein the second offset points to the stateinformation of a first data sequence of the at least one data sequence.

The method of an embodiment comprises forming a first code path throughthe data capsule using a first offset of the at least one offset. Themethod of an embodiment comprises forming a second code path through thedata capsule using a second offset of the at least one offset, whereinthe first code path and the second code path are different paths.

At least one of the first offset and the second offset of an embodimentinclude metadata, the metadata comprising context-specific metadatacorresponding to a context of the application.

The method of an embodiment comprises transferring the data capsule to apool of the plurality of pools.

The method of an embodiment comprises searching the plurality of poolsfor data capsules corresponding to the second event. The method of anembodiment comprises extracting the data capsule from the pool inresponse to the identifying of the correspondence.

The plurality of pools of an embodiment is coupled to the applicationand the at least one second application, the plurality of poolsincluding a plurality of data capsules corresponding to the applicationand the at least one second application, the plurality of poolsproviding access to the plurality of data capsules by the applicationand the at least one second application.

The plurality of pools of an embodiment provides state caching of aplurality of data capsules.

The plurality of pools of an embodiment provides linear sequencing of aplurality of data capsules.

The data structure of an embodiment is untyped.

The data structure of the data capsule of an embodiment provides aplatform-independent representation of the event data and the stateinformation.

The data structure of the data capsule of an embodiment providesplatform-independent access to the event data and the state information.

Embodiments described herein include a method comprising: dividing anapplication program into a plurality of processes; generating a portionof an output of the application program using a process of the pluralityof processes; encapsulating the portion of the output in a first datacapsule and transferring the first data capsule to at least one of aplurality of pools, wherein the plurality of pools comprise a pluralityof data capsules received from the plurality of processes; accessing theplurality of pools and retrieving an input for a second process of theplurality of processes, wherein the input is in a second data capsule ofthe plurality of data capsules; and coordinating processing among theplurality of processes using the plurality of data capsules and theplurality of pools.

Embodiments described herein include a system comprising: at least oneprocessing device, the processing device executing a plurality ofprocesses, the plurality of processes including separable programexecution contexts of a plurality of application programs, wherein eachapplication program comprises at least one process; and a plurality ofpools coupled to the at least one processing device; the at least oneprocessing device translating events of each process of the plurality ofprocesses into data capsules and transferring the data capsules into aplurality of pools, wherein a data capsule includes anapplication-independent representation of event data of an event andstate information of the process originating the data capsule; eachprocess operating as a recognizing process, the recognizing processrecognizing in the plurality of pools data capsules comprising at leastone of content that corresponds to an interactive function of therecognizing process and an identification of the recognizing process;the recognizing process retrieving recognized data capsules from theplurality of pools and executing processing appropriate to contents ofthe recognized data capsules.

Embodiments described herein include a method comprising: detecting fromgesture data a gesture made by a body, wherein the gesture data isreceived via a detector; executing a plurality of processes on aprocessing device, the plurality of processes generating events, theevents including a set of events representing the gesture; translatingthe events of each process of the plurality of processes into datacapsules; transferring the data capsules into a plurality of pools; aset of processes of the plurality of processes operating as recognizingprocesses, the recognizing processes recognizing in the plurality ofpools data capsules comprising content that corresponds to the gesture;and the recognizing processes retrieving recognized data capsules fromthe plurality of pools and generating a gesture signal from therecognized data capsules by compositing contents of the recognized datacapsules to form the gesture signal, wherein the gesture signalrepresents the gesture.

The plurality of processes of an embodiment includes separable programexecution contexts of a spatial operating application.

The gesture data of an embodiment is absolute three-space location dataof an instantaneous state of a user at a point in time and space.

The method of an embodiment comprises identifying the gesture using onlythe gesture data.

The detecting of an embodiment includes at least one of detecting alocation of the body, detecting an orientation of the body, anddetecting motion of the body.

The method of an embodiment comprises identifying the gesture, whereinthe identifying includes identifying a pose and an orientation of aportion of the body.

The detecting of an embodiment includes detecting at least one of afirst set of appendages and a second set of appendages of the body.

The detecting of an embodiment includes dynamically detecting a positionof at least one tag coupled to the body.

The detecting of an embodiment includes detecting position of a set oftags coupled to the body.

Each tag of the set of tags of an embodiment includes a pattern, whereineach pattern of each tag of the set of tags is different than anypattern of any remaining tag of the plurality of tags.

The detecting of an embodiment includes dynamically detecting andlocating a marker on the body.

The detecting of an embodiment includes detecting position of a set ofmarkers coupled to the body.

The set of markers of an embodiment form a plurality of patterns on thebody.

The detecting of an embodiment includes detecting position of aplurality of appendages of the body using a set of markers coupled toeach of the appendages.

The translating of an embodiment comprises translating information ofthe gesture to a gesture notation.

The gesture notation of an embodiment represents a gesture vocabulary,and the gesture signal comprises communications of the gesturevocabulary.

The gesture vocabulary of an embodiment represents in textual forminstantaneous pose states of kinematic linkages of the body.

The gesture vocabulary of an embodiment represents in textual form anorientation of kinematic linkages of the body.

The gesture vocabulary of an embodiment represents in textual form acombination of orientations of kinematic linkages of the body.

The gesture vocabulary of an embodiment includes a string of charactersthat represent a state of kinematic linkages of the body.

The kinematic linkage of an embodiment is at least one first appendageof the body.

The method of an embodiment comprises assigning each position in thestring to a second appendage, the second appendage connected to thefirst appendage.

The method of an embodiment comprises assigning characters of aplurality of characters to each of a plurality of positions of thesecond appendage.

The plurality of positions of an embodiment is established relative to acoordinate origin.

The method of an embodiment comprises establishing the coordinate originusing a position selected from a group consisting of an absoluteposition and orientation in space, a fixed position and orientationrelative to the body irrespective of an overall position and heading ofthe body, and interactively in response to an action of the body.

The method of an embodiment comprises assigning characters of theplurality of characters to each of a plurality of orientations of thefirst appendage.

The detecting of an embodiment comprises detecting when an extrapolatedposition of the body intersects virtual space, wherein the virtual spacecomprises space depicted on a display device coupled to the at least oneprocessing device.

The method of an embodiment comprises controlling a virtual object inthe virtual space when the extrapolated position intersects the virtualobject.

The controlling of an embodiment comprises controlling a position of thevirtual object in the virtual space in response to the extrapolatedposition in the virtual space.

The controlling of an embodiment comprises controlling attitude of thevirtual object in the virtual space in response to the gesture.

The method of an embodiment comprises controlling scaling of thedetecting and controlling to generate coincidence between virtual spaceand the physical space, wherein the virtual space comprises spacedepicted on a display, wherein the physical space comprises spaceinhabited by the body.

The method of an embodiment comprises controlling at least one virtualobject in the virtual space in response to movement of at least onephysical object in the physical space.

The method of an embodiment comprises controlling a component using thegesture signal, the component coupled to the at least one processingdevice.

Controlling the component of an embodiment comprises controlling athree-space object in six degrees of freedom simultaneously by mappingthe gesture to a three-space object.

Controlling the component of an embodiment comprises controlling athree-space object through three translational degrees of freedom andthree rotational degrees of freedom.

The three-space object of an embodiment is presented on a display devicecoupled to the at least one processing device.

The three-space object of an embodiment is a remote system coupled tothe computer.

The method of an embodiment comprises controlling movement of thethree-space object by mapping the gesture to a plurality of objecttranslations of the three-space object.

The mapping of an embodiment includes a direct mapping between thegesture and the plurality of object translations.

The mapping of an embodiment includes an indirect mapping between thegesture and the plurality of object translations.

A data capsule of an embodiment includes an application-independentrepresentation of event data of an event and state information of theprocess originating the data message.

The method of an embodiment comprises forming an interactive applicationfrom the plurality of processes by coordinating operations of each ofthe plurality of processes using the data capsules and the plurality ofpools.

The method of an embodiment comprises coordinating operations of theplurality of processes using at least one of the data capsules and theplurality of pools.

The method of an embodiment comprises dividing an application programinto a set of processes, wherein the plurality of processes includes theset of processes.

The method of an embodiment comprises a process generating an output byinteractively processing a plurality of retrieved data capsules of atleast one pool of the plurality of pools.

The plurality of processes of an embodiment includes separable programexecution contexts of a plurality of application programs, wherein eachapplication program comprises at least one process.

The method of an embodiment comprises executing the plurality ofprocesses in parallel.

The method of an embodiment comprises executing a first set of processesin parallel, and executing a second set of processes in sequence,wherein the plurality of processes includes the first set of processesand the second set of processes.

The events of an embodiment represent process inputs.

The events of an embodiment represent process outputs.

The events of an embodiment comprise a user interface event.

The events of an embodiment comprise a graphics event.

The events of an embodiment represent process state.

The state of a process of an embodiment represents the interactivefunction of the process, wherein the interactive function of the processis exposed to the plurality of processes as contents of the datacapsules.

The method of an embodiment comprises defining an applicationprogramming interface (API) of the plurality of processes by contents ofthe data capsules instead of defining the API by function calls.

The contents of the data capsules of an embodiment areapplication-independent and recognizable by the plurality of processes.

The at least one processing device of an embodiment comprises aplurality of processing devices.

At least one first set of processes of the plurality of processes of anembodiment are running under at least one first set of processingdevices of the plurality of processing devices and at least one secondset of processes of the plurality of processes are running under atleast one second set of processing devices of the plurality ofprocessing devices.

The plurality of processes of an embodiment includes a first process.

The translating of an embodiment comprises transforming events of thefirst process into at least one data sequence comprising first processevent data specifying the event and state information of the event.

The first process event data and state information of an embodiment aretype-specific data having a type corresponding to an application of thefirst process.

The translating of an embodiment comprises forming the data capsule toinclude the at least one data sequence, the data capsule having a datastructure comprising an application-independent representation of the atleast one data sequence.

The plurality of processes of an embodiment includes a second process.

The translating of an embodiment comprises transforming state changeevents of the second process into at least one data sequence comprisingsecond process event data specifying the event and state information ofthe event.

The second process event data and state information of an embodiment aretype-specific data having a type corresponding to an application of thesecond process.

The translating of an embodiment comprises forming the data capsule toinclude the at least one data sequence, the data capsule having a datastructure comprising an application-independent representation of the atleast one data sequence.

The recognizing process of an embodiment is the second process, theretrieving comprising the second process retrieving the recognized datacapsules from the plurality of pools and executing processingappropriate to contents of the recognized data capsules.

The contents of the recognized data capsules of an embodiment are datarepresenting state information of the first process.

The translating of an embodiment comprises transforming the contents ofthe recognized data capsules into at least one new data sequence, the atleast one new data sequence representing at least one of an event of thefirst process and an event of the second process.

The at least one new data sequence of an embodiment comprises event dataspecifying the event and state information of at least one of the firstprocess and the second process.

The event data and state information of the at least one of the firstprocess and the second process of an embodiment are type-specific datahaving a type corresponding to an application of the at least one of thefirst process and the second process.

The translating of an embodiment comprises forming the data capsule toinclude the at least one new data sequence, the data capsule having adata structure comprising an application-independent representation ofthe at least one new data sequence.

The plurality of processes of an embodiment uses the at least one newdata sequence.

The processing appropriate to contents of the recognized data capsulesof an embodiment comprises rendering a graphical object, wherein thegraphical object is rendered on a display of the at least one processingdevice.

The rendering of an embodiment comprises direct rendering in which theplurality of processes draw directly to a graphics layer of the at leastone processing device, wherein the plurality of pools is used forcoordination among the plurality of processes as appropriate to therendering.

The rendering of an embodiment comprises the plurality of processestransferring data capsules comprising rendering commands into theplurality of pools. The rendering of an embodiment comprises theplurality of processes retrieving the rendering commands from theplurality of pools, interpreting the rendering commands, and driving agraphics layer of the at least one processing device in response to therendering commands.

The rendering of an embodiment comprises the plurality of processesrendering to pixel buffers. The rendering of an embodiment comprises theplurality of processes transferring raw frame data into the plurality ofpools, the raw frame data resulting from the rendering to the pixelbuffers. The rendering of an embodiment comprises the plurality ofprocesses retrieving the raw frame data from the plurality of pools andcombining the raw frame data for use in driving a graphics layer of theat least one processing device.

The method of an embodiment comprises detecting an event of theplurality of processes. The method of an embodiment comprises generatingat least one data sequence comprising event data specifying the eventand state information of the event, wherein the event data and stateinformation are type-specific data having a type corresponding to anapplication of the at least one processing device. The method of anembodiment comprises forming a data capsule to include the at least onedata sequence, the data capsule having a data structure comprising anapplication-independent representation of the at least one datasequence.

The generating of the at least one data sequence of an embodimentcomprises generating a first respective data set that includes firstrespective event data. The generating of the at least one data sequenceof an embodiment comprises generating a second respective data set thatincludes second respective state information. The generating of the atleast one data sequence of an embodiment comprises forming a first datasequence to include the first respective data set and the secondrespective data set.

The generating of the first respective data set of an embodimentincludes forming the first respective data set to include identificationdata of the at least one processing device, the identification dataincluding data identifying the at least one processing device.

The generating of the at least one data sequence of an embodimentcomprises generating a first respective data set that includes firstrespective event data. The generating of the at least one data sequenceof an embodiment comprises generating a second respective data set thatincludes second respective state information. The generating of the atleast one data sequence of an embodiment comprises forming a second datasequence to include the first respective data set and the secondrespective data set.

The generating of the first respective data set of an embodimentincludes generating a first respective data set offset, wherein thefirst respective data set offset points to the first respective data setof the second data sequence.

The generating of the second respective data set of an embodimentincludes generating a second respective data set offset, wherein thesecond respective data set offset points to the second respective dataset of the second data sequence.

The first respective data set of an embodiment is a description list,the description list including a description of the data.

The event data of an embodiment is a tagged byte-sequence representingtyped data.

The event data of an embodiment includes a type header and atype-specific data layout.

The state information of an embodiment is a tagged byte-sequencerepresenting typed data.

The state information of an embodiment includes a type header and atype-specific data layout.

The method of an embodiment comprises generating at least one offset.The method of an embodiment comprises forming the data capsule toinclude the at least one offset.

The method of an embodiment comprises generating a first offset having afirst variable length, wherein the first offset points to the event dataof a first data sequence of the at least one data sequence.

The method of an embodiment comprises generating a second offset havinga second variable length, wherein the second offset points to the stateinformation of a first data sequence of the at least one data sequence.

The method of an embodiment comprises forming a first code path throughthe data capsule using a first offset of the at least one offset. Themethod of an embodiment comprises forming a second code path through thedata capsule using a second offset of the at least one offset, whereinthe first code path and the second code path are different paths.

At least one of the first offset and the second offset of an embodimentinclude metadata, the metadata comprising context-specific metadatacorresponding to a context of the application.

The method of an embodiment comprises generating a header that includesa length of the data capsule. The method of an embodiment comprisesforming the data capsule to include the header.

The method of an embodiment comprises transferring the data capsule to apool of the plurality of pools.

The method of an embodiment comprises detecting a second event of the atleast one processing device. The method of an embodiment comprisessearching the plurality of pools for data capsules corresponding to thesecond event.

The method of an embodiment comprises identifying a correspondencebetween the data capsule and the second event. The method of anembodiment comprises extracting the data capsule from the pool inresponse to the identifying. The method of an embodiment comprisesexecuting on behalf of the at least one processing device a processingoperation corresponding to the second event in response to contents ofthe data capsule, wherein the at least one processing device correspondsto an application of a first type and a second application of a secondtype.

The plurality of pools of an embodiment is coupled to a plurality ofapplications, the plurality of pools including a plurality of datacapsules corresponding to the plurality of applications, the pluralityof pools providing access to the plurality of data capsules by theplurality of applications, wherein at least two applications of theplurality of applications are different applications.

The plurality of pools of an embodiment provides state caching of aplurality of data capsules.

The plurality of pools of an embodiment provides linear sequencing of aplurality of data capsules.

The data structure of an embodiment is untyped.

The data structure of the data capsule of an embodiment provides aplatform-independent representation of the event data and the stateinformation.

The data structure of the data capsule of an embodiment providesplatform-independent access to the event data and the state information.

The transferring of an embodiment comprises transferring the datacapsule from a first application having a first application type to atleast one second application having at least one second applicationtype, wherein the first application type is different than the secondapplication type, wherein the generating of the at least one datasequence was executed by the first application, the method comprisingmaintaining intact the at least one data sequence of the data capsuleduring the transferring.

The method of an embodiment comprises using the at least one datasequence during operations of the second application.

The method of an embodiment comprises generating a first data set thatincludes event data and identification data of a source device of the atleast one processing device, the device event data including dataspecifying an event registered by the source device, the identificationdata including data identifying the source device.

The method of an embodiment comprises generating a second data set thatincludes a full set of state information of the event, wherein each ofthe first data set and the second data set comprise typed data bundlesin a type-specific data layout.

The translating of an embodiment comprises encapsulating the first dataset and the second data set by forming a data capsule to include thefirst data set and the second data set, wherein the data capsule has adata structure comprising an application-independent representation ofthe at least one data sequence.

The method of an embodiment comprises detecting an event of a firstprocessing device running under an application of a first type. Themethod of an embodiment comprises generating data sequences comprisingevent data of the first processing device, the event data specifying theevent and state information of the event, wherein the event data andstate information are type-specific data having a type corresponding tothe application. The method of an embodiment comprises forming a datacapsule to include the data sequences, the data capsule having a datastructure comprising an application-independent representation of thedata sequences. The method of an embodiment comprises detecting a secondevent of a second processing device running under at least one secondapplication having at least one second type, wherein the second type isdifferent from the first type, wherein the at least one processingdevice comprises the first processing device and the second processingdevice. The method of an embodiment comprises identifying acorrespondence between the data capsule and the second event. The methodof an embodiment comprises executing an operation in response to thesecond event using contents of the data sequences of the data capsule.

The generating of the data sequences of an embodiment comprisesgenerating a first data set that includes the event data. The generatingof the data sequences of an embodiment comprises generating a seconddata set that includes the state information. The generating of the datasequences of an embodiment comprises forming a first data sequence toinclude the first data set and the second data set.

The event data of an embodiment is a tagged byte-sequence representingtyped data.

The event data of an embodiment includes a type header and atype-specific data layout.

The state information of an embodiment is a tagged byte-sequencerepresenting typed data.

The state information of an embodiment includes a type header and atype-specific data layout.

The method of an embodiment comprises generating at least one offset.The method of an embodiment comprises forming the data capsule toinclude the at least one offset.

The method of an embodiment comprises generating a first offset having afirst variable length, wherein the first offset points to the event dataof a first data sequence of the at least one data sequence. The methodof an embodiment comprises generating a second offset having a secondvariable length, wherein the second offset points to the stateinformation of a first data sequence of the at least one data sequence.

The method of an embodiment comprises forming a first code path throughthe data capsule using a first offset of the at least one offset. Themethod of an embodiment comprises forming a second code path through thedata capsule using a second offset of the at least one offset, whereinthe first code path and the second code path are different paths.

At least one of the first offset and the second offset of an embodimentinclude metadata, the metadata comprising context-specific metadatacorresponding to a context of the application.

The method of an embodiment comprises transferring the data capsule to apool of the plurality of pools.

The method of an embodiment comprises searching the plurality of poolsfor data capsules corresponding to the second event. The method of anembodiment comprises extracting the data capsule from the pool inresponse to the identifying of the correspondence.

The plurality of pools of an embodiment is coupled to the applicationand the at least one second application, the plurality of poolsincluding a plurality of data capsules corresponding to the applicationand the at least one second application, the plurality of poolsproviding access to the plurality of data capsules by the applicationand the at least one second application.

The plurality of pools of an embodiment provides state caching of aplurality of data capsules.

The plurality of pools of an embodiment provides linear sequencing of aplurality of data capsules.

The data structure of an embodiment is untyped.

The data structure of the data capsule of an embodiment provides aplatform-independent representation of the event data and the stateinformation.

The data structure of the data capsule of an embodiment providesplatform-independent access to the event data and the state information.

Embodiments described herein include a method comprising: executing aplurality of processes on a processing device, the plurality ofprocesses including separable program execution contexts of a pluralityof application programs, wherein each application program comprises atleast one process; translating events of each process of the pluralityof processes into data messages, wherein a data message includes anapplication-independent representation of event data of an event andstate information of the process originating the data message;transferring the data messages into at least one pool of a plurality ofpools; coordinating among the processes, the coordinating including eachprocess of the plurality of processes coordinating with peer processesof the plurality of processes by retrieving from the plurality of poolsthe state information of the peer processes; and generating an output ofthe plurality of processes by interactively combining a set of datamessages of at least one pool of the plurality of pools.

Embodiments described herein include a system comprising: a detector forreceiving gesture data that represents a gesture made by a body; and aprocessor coupled to the detector, the processor automatically detectingthe gesture from the gesture data, the processor executing a pluralityof processes, the plurality of processes generating events that includea set of events representing the gesture, the processor translating theevents of each process of the plurality of processes into data capsules,the processor transferring the data capsules into a plurality of pools,wherein a set of processes of the plurality of processes function asrecognizing processes, the recognizing processes recognizing in theplurality of pools data capsules comprising content that corresponds tothe gesture, the recognizing processes retrieving recognized datacapsules from the plurality of pools and generating a gesture signalfrom the recognized data capsules by compositing contents of therecognized data capsules to form the gesture signal, wherein the gesturesignal represents the gesture.

The systems and methods described herein include and/or run under and/orin association with a processing system. The processing system includesany collection of processor-based devices or computing devices operatingtogether, or components of processing systems or devices, as is known inthe art. For example, the processing system can include one or more of aportable computer, portable communication device operating in acommunication network, and/or a network server. The portable computercan be any of a number and/or combination of devices selected from amongpersonal computers, cellular telephones, personal digital assistants,portable computing devices, and portable communication devices, but isnot so limited. The processing system can include components within alarger computer system.

The processing system of an embodiment includes at least one processorand at least one memory device or subsystem. The processing system canalso include or be coupled to at least one database. The term“processor” as generally used herein refers to any logic processingunit, such as one or more central processing units (CPUs), digitalsignal processors (DSPs), application-specific integrated circuits(ASIC), etc. The processor and memory can be monolithically integratedonto a single chip, distributed among a number of chips or components ofa host system, and/or provided by some combination of algorithms. Themethods described herein can be implemented in one or more of softwarealgorithm(s), programs, firmware, hardware, components, circuitry, inany combination.

System components embodying the systems and methods described herein canbe located together or in separate locations. Consequently, systemcomponents embodying the systems and methods described herein can becomponents of a single system, multiple systems, and/or geographicallyseparate systems. These components can also be subcomponents orsubsystems of a single system, multiple systems, and/or geographicallyseparate systems. These components can be coupled to one or more othercomponents of a host system or a system coupled to the host system.

Communication paths couple the system components and include any mediumfor communicating or transferring files among the components. Thecommunication paths include wireless connections, wired connections, andhybrid wireless/wired connections. The communication paths also includecouplings or connections to networks including local area networks(LANs), metropolitan area networks (MANs), wide area networks (WANs),proprietary networks, interoffice or backend networks, and the Internet.Furthermore, the communication paths include removable fixed mediumslike floppy disks, hard disk drives, and CD-ROM disks, as well as flashRAM, Universal Serial Bus (USB) connections, RS-232 connections,telephone lines, buses, and electronic mail messages.

Unless the context clearly requires otherwise, throughout thedescription, the words “comprise,” “comprising,” and the like are to beconstrued in an inclusive sense as opposed to an exclusive or exhaustivesense; that is to say, in a sense of “including, but not limited to.”Words using the singular or plural number also include the plural orsingular number respectively. Additionally, the words “herein,”“hereunder,” “above,” “below,” and words of similar import refer to thisapplication as a whole and not to any particular portions of thisapplication. When the word “or” is used in reference to a list of two ormore items, that word covers all of the following interpretations of theword: any of the items in the list, all of the items in the list and anycombination of the items in the list.

The above description of embodiments is not intended to be exhaustive orto limit the systems and methods described to the precise formdisclosed. While specific embodiments and examples are described hereinfor illustrative purposes, various equivalent modifications are possiblewithin the scope of other systems and methods, as those skilled in therelevant art will recognize. The teachings provided herein can beapplied to other processing systems and methods, not only for thesystems and methods described above.

The elements and acts of the various embodiments described above can becombined to provide further embodiments. These and other changes can bemade to the embodiments in light of the above detailed description.

In general, in the following claims, the terms used should not beconstrued to limit the embodiments to the specific embodiments disclosedin the specification and the claims, but should be construed to includeall systems that operate under the claims. Accordingly, the embodimentsare not limited by the disclosure herein, but instead the scope of theembodiments is to be determined entirely by the claims.

While certain aspects of the embodiments are presented below in certainclaim forms, the inventors contemplate the various aspects of theembodiments in any number of claim forms. Accordingly, the inventorsreserve the right to add additional claims after filing the applicationto pursue such additional claim forms for other aspects of theembodiments.

1. (canceled)
 2. A method comprising: displaying a multi-user virtualspace that includes frame data of a plurality of processes across aplurality of display devices by using a plurality of processing devices;generating a spatial user input data capsule for each user of a processincluded in the virtual space, and transferring each spatial user inputdata capsule to a user input repository that is shared across theplurality of processes, each spatial user input data capsule includingspatial user input data of a three-space coordinate system; and at leastone process of the plurality of processes accessing the user inputrepository and updating frame data of the virtual space based on atleast one spatial user input data capsule included in the user inputrepository.
 3. The method of claim 2, wherein at least one process ofthe plurality of processes accessing the user input repository andupdating frame data of the virtual space comprises: at least one pointerprocess: determining a location of a pointer for each user of thevirtual space by accessing at least one spatial user input data capsulefrom the user input repository, and updating frame data of the virtualspace to control the plurality of display devices to display eachpointer based on each determined location, wherein each spatial userinput data capsule used by a pointer process defines a position of apointer object with respect to one of the plurality of display devices.4. The method of claim 3, wherein each of a plurality of pointerprocesses updates frame data for a different set of the plurality ofdisplay devices, and wherein each pointer process recognizes spatialuser input data capsules that correspond to a display device controlledby the pointer process in the user input repository and retrieves eachrecognized spatial user input data capsule from the user inputrepository.
 5. The method of claim 4, wherein at least one process ofthe plurality of processes accessing the user input repository andupdating frame data of the virtual space comprises: at least oneapplication process: updating frame data of the application by accessingat least one spatial user input data capsule from the user inputrepository, generating a frame data capsule that includes the updatedframe data, and transferring the frame data capsule to a frame datarepository, wherein frame data included in data capsules in the framedata repository are rendered to generate display output for display bythe plurality of display devices.
 6. The method of claim 5, wherein eachof a plurality of application processes recognizes spatial user inputdata capsules that correspond to the application process in the userinput repository and retrieves each recognized spatial user input datacapsule from the user input repository.
 7. The method of claim 4,wherein at least one process of the plurality of processes accessing theuser input repository and updating frame data of the virtual spacecomprises: at least one pointer process: generating at least one pointerdata capsule that includes pointer data of at least one displayedpointer, and transferring each generated pointer data capsule to acoordination repository, wherein pointer data included in pointer datacapsules in the coordination data repository are accessible by at leastone application process; and at least one application process: updatingframe data of the application by accessing at least one pointer datacapsule from the coordination data repository, generating a frame datacapsule that includes the updated frame data, and transferring the framedata capsule to a frame data repository, wherein frame data included indata capsules in the frame data repository are rendered to generatedisplay output for display by the plurality of display devices.
 8. Themethod of claim 7, wherein each of a plurality of application processesrecognizes pointer data capsules that correspond to the applicationprocess in the coordination data repository and retrieves eachrecognized pointer data capsule from the coordination data repository.9. The method of claim 2, further comprising: determining dimensions,orientations and positions of each of the plurality of display devicesin physical space; generating at least one data capsule identifying thedetermined dimensions, orientations and positions of each of theplurality of display devices; and transferring each data capsuleidentifying the determined dimensions, orientations and positions ofeach of the plurality of display devices to a display device repositoryaccessible by each of the plurality of processing devices.
 10. Themethod of claim 9, wherein displaying a multi-user virtual spacecomprises: each processing device of the plurality of processing devicescontrolling a different set of one or more of the plurality of displaydevices to display frame outputs of a rendering process of theprocessing device, each rendering process generating a respective set offrame outputs of the virtual space.
 11. The method of claim 10, whereinat least one of the plurality of processing devices recognizes a spatialuser input data capsule in the user input repository, retrieves therecognized spatial user input data capsule, and performs rendering basedon the retrieved spatial user input data capsule.
 12. The method ofclaim 10, further comprising: at least one of the plurality ofprocessing devices retrieving at least one data capsule identifying thedetermined dimensions, orientations and positions of at least one of thepluralities of display devices from the display device repository, andusing each data capsule retrieved form the display device repository toperform rendering to dynamically map the physical space in which thedisplay devices are located as a projection into the virtual space. 13.A method comprising: a plurality of processing devices controlling aplurality of display devices, each processing device controlling adifferent set of one or more display devices to display frame outputs ofa rendering process of the processing device, each rendering processgenerating a respective set of frame outputs of a first application todepict a virtual space of the first application across the plurality ofdisplay devices; generating a spatial user input data capsule andtransferring the spatial user input data capsule to a user inputrepository to simultaneously control rendering of frame outputs of thefirst application by at least two of the plurality of processingdevices, and each of the at least two of the plurality of processingdevices recognizing the spatial user input data capsule in the userinput repository, retrieving the recognized spatial user input datacapsule, and performing rendering based on the user input data capsule.14. The method of claim 13, wherein the spatial user input data capsuleincludes spatial user input data of a three-space coordinate system. 15.The method of claim 14, wherein a user input device generates thespatial user input data capsule.
 16. The method of claim 14, wherein agestural process generates the spatial user input data capsule.
 17. Asystem comprising: a plurality of display devices; a plurality ofprocesses running on a plurality of processing devices, the plurality ofprocesses configured to control the plurality of display devices todisplay a multi-user virtual space that includes frame data of theplurality of processes, each processing device configured to control adifferent set of one or more of the plurality of display devices todisplay outputs of a rendering process of the processing device, eachrendering process generating a respective set of frame outputs for theframe data of the virtual space; a user input repository configured toreceive spatial user input data capsules and share spatial user inputdata capsules across the plurality of processes, each spatial user inputdata capsule including spatial user input data of a three-spacecoordinate system for at least one user of a process of the plurality ofprocesses; and a frame data repository configured to receive frame datacapsules and share frame data capsules across the plurality ofprocesses, wherein frame data included in frame data capsules in theframe data repository are rendered by at least one of the plurality ofprocesses to generate display output for display by the plurality ofdisplay devices.
 18. The system of claim 17, further comprising: acoordination repository configured to receive pointer data capsules andshare pointer data capsules across the plurality of processes, eachpointer data capsule including pointer data generated from at least onespatial user input data capsule of the user input repository.
 19. Thesystem of claim 18, further comprising: a display data repositoryconfigured to receive display data capsules and share display devicedata capsules across the plurality of processes, wherein data includedin each display device data capsule includes information identifyingdimensions, orientations and positions of each of the plurality ofdisplay devices.
 20. The system of claim 19, further comprising a userinput device that is configured to generate a spatial user input datacapsule.
 21. The system of claim 19, further comprising a gesturalprocess that is configured to generate a spatial user input datacapsule.